01 · Operator
Who operates Yahoo! Slurp
- Company
- Yahoo
- Official docs
- help.yahoo.com
02 · Behavior
What Yahoo! Slurp does
Yahoo describes Slurp as its search robot for crawling and indexing pages. It also collects content from partner sites for Yahoo News, Yahoo Finance and Yahoo Sports. Yahoo states that Slurp obeys robots.txt.
03 · Impact
Why Yahoo! Slurp matters for your site
If you allow it
- May support content in Yahoo properties
- Honors robots.txt
If you block it
- Cloudflare Radar describes it in the past tense as Yahoo's search crawler
- Its user agent is often spoofed, so check IPs
04 · Allow
How to allow Yahoo! Slurp
robots.txt
User-agent: Slurp
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Slurp")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Slurp
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Slurp") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Slurp [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Yahoo! Slurp
Yahoo! Slurp follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: Slurp
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Slurp")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Slurp
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Slurp") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Slurp [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Yahoo! Slurp
The user agent published by the operator:
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)07 · Verification
Is it really Yahoo! Slurp?
Yahoo! Slurp’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Yahoo! Slurp
What is Yahoo! Slurp?
Slurp is Yahoo's crawler for indexing web pages. It also collects partner content for Yahoo News, Finance and Sports.
How do I block Yahoo! Slurp?
Yahoo states that Slurp obeys robots.txt. Add a group with User-agent: Slurp and your Disallow rules.
Last reviewed Oct 7, 2026