01 · Operator
Who operates ClaudeBot
- Company
- Anthropic
- Type
- AI training crawler
- Official docs
- support.claude.com
02 · Behavior
What ClaudeBot does
ClaudeBot crawls the web automatically to collect content that could contribute to model training. It is not triggered by Claude users. Restricting ClaudeBot signals that the site's future content should be excluded from Anthropic's training datasets.
03 · Impact
Why ClaudeBot matters for your site
If you allow it
- Your content can inform future Claude models
- Blocking it does not affect Claude search or user fetches (separate bots)
If you block it
- No direct referral traffic from training crawls
- Keeps future content out of Anthropic model training
- Saves crawl bandwidth
04 · Allow
How to allow ClaudeBot
robots.txt
User-agent: ClaudeBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "ClaudeBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: ClaudeBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "ClaudeBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block ClaudeBot
ClaudeBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: ClaudeBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "ClaudeBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: ClaudeBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "ClaudeBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for ClaudeBot
The operator does not publish a full user agent string. Requests contain the token “ClaudeBot”.
07 · Verification
Is it really ClaudeBot?
Anyone can copy a user agent. Real ClaudeBot requests come from the IP ranges published at claude.com/crawling/bots.json.
FAQ
Questions about ClaudeBot
Does ClaudeBot respect robots.txt?
Yes. Anthropic states its bots honor robots.txt directives and also support Crawl-delay.
Does blocking ClaudeBot remove my site from Claude search?
No. Claude search uses Claude-SearchBot and user fetches use Claude-User, each with its own robots.txt token.
How do I slow ClaudeBot down instead of blocking it?
Add a Crawl-delay line under "User-agent: ClaudeBot" in robots.txt; Anthropic documents support for it.
Last reviewed Oct 6, 2026