01 · Operator
Who operates Meta-WebIndexer
- Company
- Meta
- Type
- AI search crawler
- Official docs
- developers.facebook.com
02 · Behavior
What Meta-WebIndexer does
Meta-WebIndexer navigates the web to improve the quality of Meta AI search results. Meta analyzes the content to make Meta AI answers more relevant and accurate. Meta says allowing it helps Meta AI cite and link to your pages.
03 · Impact
Why Meta-WebIndexer matters for your site
If you allow it
- Meta AI can cite and link to your content
- Controlled through robots.txt
- Requests come from Meta's own network
If you block it
- Content is analyzed for Meta AI answers
- Adds crawl load
04 · Allow
How to allow Meta-WebIndexer
robots.txt
User-agent: meta-webindexer
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-webindexer")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-webindexer
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "meta\-webindexer") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} meta\-webindexer [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Meta-WebIndexer
Meta-WebIndexer follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: meta-webindexer
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-webindexer")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-webindexer
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "meta\-webindexer") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta\-webindexer [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Meta-WebIndexer
The user agent published by the operator:
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)07 · Verification
Is it really Meta-WebIndexer?
Meta-WebIndexer’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Meta-WebIndexer
What is meta-webindexer?
It is Meta's crawler for Meta AI search. It helps Meta AI answer questions and cite web sources.
How do I verify Meta-WebIndexer IPs?
Check that the IP belongs to Meta's AS32934. Meta suggests: whois -h whois.radb.net -- '-i origin AS32934' | grep ^route.
Is Meta-WebIndexer the same as Meta-ExternalAgent?
No. Meta lists them as separate crawlers with separate robots.txt tokens.
Last reviewed Oct 7, 2026