01 · Operator
Who operates Twitterbot
- Company
- X Corp. (formerly Twitter)
- Type
- Social preview bot
- Official docs
- web-wp.archive.org
02 · Behavior
What Twitterbot does
When a link is posted, Twitterbot requests the page and reads its card markup. It also fetches the referenced images. Twitter's documentation states that card content is cached for 7 days after a link is first posted.
03 · Impact
Why Twitterbot matters for your site
If you allow it
- Shared links show a title, description and image on X
- Blocking it removes cards and thumbnails from your links
If you block it
- You do not want your pages previewed on X
- You want to limit requests from social platforms
04 · Allow
How to allow Twitterbot
robots.txt
User-agent: Twitterbot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Twitterbot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Twitterbot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Twitterbot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Twitterbot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Twitterbot
Twitterbot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: Twitterbot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Twitterbot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Twitterbot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Twitterbot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Twitterbot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Twitterbot
The user agent published by the operator:
Twitterbot/1.007 · Verification
Is it really Twitterbot?
Twitterbot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Twitterbot
Does Twitterbot respect robots.txt?
Yes. Twitter's card documentation states that pages or images blocked by robots.txt will not show a card or thumbnail.
How do I allow Twitterbot but block other crawlers?
Add a group with User-agent: Twitterbot and an empty Disallow line. Then add User-agent: * with Disallow: / for all other bots.
How long does X cache link previews?
Twitter's documentation states that card content is cached for 7 days after the link is first posted.
Last reviewed Oct 7, 2026