01 · Operator
Who operates Meta-ExternalFetcher
- Company
- Meta
- Type
- AI assistant fetcher
- Official docs
- developers.facebook.com
02 · Behavior
What Meta-ExternalFetcher does
Meta-ExternalFetcher loads single URLs in response to user requests. Meta uses it to evaluate and improve agentic AI, including helping AI navigate websites to complete tasks for users. Meta states it may bypass robots.txt because fetches are user-initiated.
03 · Impact
Why Meta-ExternalFetcher matters for your site
If you allow it
- Meta AI users can access your pages on request
- Fetches are tied to individual user actions
If you block it
- It may bypass robots.txt rules
- Your content may be used to improve Meta's AI products
- No IP list or reverse DNS method is published for verification
04 · Allow
How to allow Meta-ExternalFetcher
robots.txt
User-agent: meta-externalfetcher
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-externalfetcher")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-externalfetcher
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "meta\-externalfetcher") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} meta\-externalfetcher [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Meta-ExternalFetcher
Meta-ExternalFetcher does not always follow robots.txt. Add a server or CDN rule if you need a hard block.
robots.txt
User-agent: meta-externalfetcher
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-externalfetcher")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-externalfetcher
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "meta\-externalfetcher") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta\-externalfetcher [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Meta-ExternalFetcher
The user agent published by the operator:
Mozilla/5.0 (compatible; Meta-ExternalFetcher/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)07 · Verification
Is it really Meta-ExternalFetcher?
Meta-ExternalFetcher’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Meta-ExternalFetcher
What is Meta-ExternalFetcher?
It is a Meta fetcher that retrieves individual links when a user requests them, used for Meta's agentic AI features.
Does Meta-ExternalFetcher respect robots.txt?
Not reliably. Meta states it may bypass robots.txt because the fetches are user-initiated. The documented token is meta-externalfetcher.
How long do robots.txt changes take to apply to Meta crawlers?
Meta states that robots.txt changes can take up to 24 hours to take effect because of caching.
Last updated Oct 19, 2026