01 · Operator
Who operates Screaming Frog SEO Spider
- Company
- Screaming Frog
- Type
- SEO tool crawler
- Official docs
- screamingfrog.co.uk
02 · Behavior
What Screaming Frog SEO Spider does
It crawls a site to find technical SEO issues such as broken links and redirects. It runs on the user's own machine, so the IP is that user's network. It obeys robots.txt by default. Licensed users can ignore robots.txt, change the user agent and render JavaScript.
03 · Impact
Why Screaming Frog SEO Spider matters for your site
If you allow it
- Your own team or agency may use it for audits
- It obeys robots.txt by default
If you block it
- Anyone can crawl your site with it
- Users can switch off robots.txt and change the user agent
- Fast crawls can load your server
04 · Allow
How to allow Screaming Frog SEO Spider
robots.txt
# Screaming Frog SEO Spider has no documented robots.txt token and does not read robots.txt.
# Use a Cloudflare, nginx or Apache rule instead.Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Screaming Frog SEO Spider")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# Screaming Frog SEO Spider has no documented robots.txt token and does not read robots.txt.
# Use a Cloudflare, nginx or Apache rule instead.nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Screaming Frog SEO Spider") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Screaming Frog SEO Spider [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Screaming Frog SEO Spider
Screaming Frog SEO Spider does not always follow robots.txt. Add a server or CDN rule if you need a hard block.
robots.txt
# Screaming Frog SEO Spider has no documented robots.txt token and does not read robots.txt.
# Use a Cloudflare, nginx or Apache rule instead.Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Screaming Frog SEO Spider")
Action: BlockWordPress
# Screaming Frog SEO Spider has no documented robots.txt token and does not read robots.txt.
# Use a Cloudflare, nginx or Apache rule instead.nginx
# In your server { } block:
if ($http_user_agent ~* "Screaming Frog SEO Spider") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Screaming Frog SEO Spider [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Screaming Frog SEO Spider
| User agent | Share | Last active | Status |
|---|---|---|---|
Screaming Frog SEO Spider/24.3 |
84% | Yesterday | Unverified |
Screaming Frog SEO Spider/16.7 |
13% | 11 days ago | Unverified |
Screaming Frog SEO Spider/20.2 |
3.2% | 21 days ago | Unverified |
07 · Verification
Is it really Screaming Frog SEO Spider?
Screaming Frog SEO Spider’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Screaming Frog SEO Spider
Can I verify Screaming Frog by IP address?
No. The SEO Spider runs on the user's own computer, so requests come from that user's network.
Does Screaming Frog obey robots.txt?
By default it does. Paid users can choose to ignore robots.txt, so robots.txt alone may not stop it.
Can Screaming Frog pretend to be Googlebot?
Yes. It has built-in user agent presets for Googlebot, Bingbot and browsers. Verify any Googlebot claim with Google's IP list.
Last updated Oct 14, 2026