01 · Operator
Who operates Claude-SearchBot
- Company
- Anthropic
- Type
- AI search crawler
- Official docs
- support.claude.com
02 · Behavior
What Claude-SearchBot does
Claude-SearchBot crawls web content automatically to improve the relevance and accuracy of Claude's search responses. It is not triggered by individual user requests. Anthropic states that disabling it prevents your content from being indexed for search.
03 · Impact
Why Claude-SearchBot matters for your site
If you allow it
- Your pages can be indexed for Claude's search answers
- Anthropic says blocking may reduce visibility and accuracy in user search results
If you block it
- Removes your content from Claude's search index
- Reduces crawl load if you do not want AI search exposure
04 · Allow
How to allow Claude-SearchBot
robots.txt
User-agent: Claude-SearchBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Claude-SearchBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Claude-SearchBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Claude\-SearchBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Claude\-SearchBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Claude-SearchBot
Claude-SearchBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: Claude-SearchBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Claude-SearchBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Claude-SearchBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Claude\-SearchBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Claude\-SearchBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Claude-SearchBot
The operator does not publish a full user agent string. Requests contain the token “Claude-SearchBot”.
07 · Verification
Is it really Claude-SearchBot?
Anyone can copy a user agent. Real Claude-SearchBot requests come from the IP ranges published at claude.com/crawling/bots.json.
FAQ
Questions about Claude-SearchBot
What is the difference between Claude-SearchBot and ClaudeBot?
Claude-SearchBot indexes content to improve Claude's search results, while ClaudeBot collects data that may be used for model training.
What happens if I block Claude-SearchBot?
Anthropic says your content will not be indexed for search, which may reduce your site's visibility and accuracy in Claude's search results.
How do I verify Claude-SearchBot requests?
Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json.
Last reviewed Oct 6, 2026