01 · Operator
Who operates FAST-WebCrawler
02 · Behavior
What FAST-WebCrawler does
It crawled web pages for the AlltheWeb index. Overture bought FAST's web search unit in 2003, and Yahoo then bought Overture. Its user agent names Overture as the contact.
03 · Impact
Why FAST-WebCrawler matters for your site
If you allow it
- None today; the search engine no longer exists
If you block it
- The service shut down in 2011
- Current hits are likely spoofed
04 · Allow
How to allow FAST-WebCrawler
robots.txt
User-agent: FAST-WebCrawler
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "FAST-WebCrawler")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: FAST-WebCrawler
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "FAST\-WebCrawler") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} FAST\-WebCrawler [NC]
# RewriteRule .* - [F,L]05 · Block
How to block FAST-WebCrawler
Start with robots.txt. If FAST-WebCrawler keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: FAST-WebCrawler
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "FAST-WebCrawler")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: FAST-WebCrawler
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "FAST\-WebCrawler") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} FAST\-WebCrawler [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for FAST-WebCrawler
The user agent published by the operator:
FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)07 · Verification
Is it really FAST-WebCrawler?
FAST-WebCrawler’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about FAST-WebCrawler
Is FAST-WebCrawler still active?
Its search engine, AlltheWeb, was shut down by Yahoo on April 4, 2011. Current requests with this name are likely spoofed.
Should I block FAST-WebCrawler?
Yes, blocking it has no downside. The search engine it served no longer exists.
Who operated FAST-WebCrawler?
FAST Search & Transfer built AlltheWeb. Overture and later Yahoo owned it after 2003.
Last reviewed Oct 7, 2026