01 · Operator
Who operates ExaSearchBot
- Company
- Exa
- Type
- AI search crawler
- Official docs
- crawler.exa.ai
02 · Behavior
What ExaSearchBot does
ExaSearchBot crawls public web pages so Exa can index them. Exa uses the index to connect people and applications to content and send them back to the source site. The crawler does not bypass logins, paywalls or CAPTCHAs and submits no forms. Its requests are signed with HTTP Message Signatures (RFC 9421).
03 · Impact
Why ExaSearchBot matters for your site
If you allow it
- Your pages can appear in Exa search and in AI apps built on Exa
- Exa states it sends users back to your site
- Requests can be verified cryptographically
If you block it
- High crawl volume can add server load
- You may not want your content in AI retrieval products
04 · Allow
How to allow ExaSearchBot
robots.txt
User-agent: ExaSearchBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "ExaSearchBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: ExaSearchBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "ExaSearchBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} ExaSearchBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block ExaSearchBot
ExaSearchBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: ExaSearchBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "ExaSearchBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: ExaSearchBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "ExaSearchBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ExaSearchBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for ExaSearchBot
The user agent published by the operator:
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)07 · Verification
Is it really ExaSearchBot?
ExaSearchBot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about ExaSearchBot
What is ExaSearchBot?
ExaSearchBot is the crawler of the Exa search engine. It indexes public pages for search and retrieval.
Does ExaSearchBot respect robots.txt?
Yes. Exa states that ExaSearchBot respects the Robots Exclusion Protocol. Use the token ExaSearchBot in robots.txt.
How can I verify ExaSearchBot requests?
Exa signs requests per RFC 9421. Its Ed25519 public keys are published at crawler.exa.ai/.well-known/http-message-signatures-directory.
Last reviewed Oct 7, 2026