Log in
Sign up with Google

AI training crawler · Meta

Meta-ExternalAgent is Meta’s AI training crawler

Meta-ExternalAgent is Meta's crawler for uses such as training AI models and improving products by indexing content directly.

Verifiable operator Respects robots.txt

Our take

Your call

It mainly collects content for AI training and product indexing, so allowing it is a content-licensing decision.

User agent

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Activity

Meta-ExternalAgent is crawling more than last month

Requests per site per dayLAST 28 DAYS
Oct 19Oct 28Nov 6Nov 15
What it crawls30 DAYS
  • Content pages 99%
  • Not found (404) 0.9%
  • Other 0.1%

01 · Operator

Who operates Meta-ExternalAgent

Company
Meta
Official docs
developers.facebook.com

02 · Behavior

What Meta-ExternalAgent does

Meta-ExternalAgent crawls the web automatically for use cases such as training foundation AI models or indexing content for Meta products. It is not triggered by users; user-requested fetches use Meta-ExternalFetcher. Meta states it respects robots.txt.

03 · Impact

Why Meta-ExternalAgent matters for your site

If you allow it

  • Your content can inform Meta AI models and products
  • Meta says it follows robots.txt, so you can limit it to parts of your site

If you block it

  • Keeps your content out of Meta AI training
  • No documented direct referral traffic
  • Saves crawl bandwidth

04 · Allow

How to allow Meta-ExternalAgent

robots.txt

User-agent: meta-externalagent
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-externalagent")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-externalagent
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "meta\-externalagent") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} meta\-externalagent [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block Meta-ExternalAgent

Meta-ExternalAgent follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.

robots.txt

User-agent: meta-externalagent
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "meta-externalagent")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: meta-externalagent
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "meta\-externalagent") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta\-externalagent [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for Meta-ExternalAgent

User agentShareLast activeStatus
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Sa… 58% 3 days ago Unverified
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.… 28% 3 days ago Unverified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Sa… 8.4% Yesterday Verified
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.… 4.1% Yesterday Verified
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.3… 0.5% Yesterday Verified
Mozilla/5.0 (compatible; Meta-ExternalAgent/1.0; +https://developers.facebook.com/docs/sharing/webma… 0.3% 7 days ago Unverified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Sa… 0.1% Yesterday Verified
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) 0% 2 days ago Verified
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; me… 0% Yesterday Verified
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) 0% 4 days ago Unverified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Sa… 0% Yesterday Verified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Sa… 0% Yesterday Verified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Sa… 0% 2 days ago Verified
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.7632.6… 0% 2 days ago Verified
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.3… 0% 3 days ago Unverified

07 · Verification

Is it really Meta-ExternalAgent?

Meta-ExternalAgent’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

IP addresses we saw from verified Meta-ExternalAgent requests

IP rangeShareLast active
57.141.22.47 13% Yesterday
57.141.22.10 13% Yesterday
57.141.0.8 5.3% 10 days ago
57.141.0.51 4.9% 10 days ago
57.141.0.26 4.9% 10 days ago
57.141.0.45 4.8% 10 days ago
57.141.0.52 4.7% 10 days ago
57.141.0.30 4.7% 10 days ago
57.141.0.34 4.6% 10 days ago
57.141.0.24 4.6% 10 days ago

Top 10 of 38 ranges. Always check against the operator’s current list; ranges change.

FAQ

Questions about Meta-ExternalAgent

Does blocking Meta-ExternalAgent affect link previews on Facebook?

Link previews are fetched by a different crawler, facebookexternalhit, which has its own robots.txt token.

How do I verify Meta-ExternalAgent?

Meta does not publish a JSON list; it says to get its current IP ranges by querying routes for AS32934, e.g. whois -h whois.radb.net -- '-i origin AS32934'.

How do I block Meta-ExternalAgent?

Add "User-agent: meta-externalagent" followed by "Disallow: /" to robots.txt; Meta states the crawler respects these rules.

Last updated Nov 15, 2026

Your site

See which pages Meta-ExternalAgent crawls on your website

Log Hero reads your server logs and shows every AI bot, every page, every day.

Sign up with Google

Free during early access