01 · Operator
Who operates Googlebot
- Company
- Official docs
- developers.google.com
02 · Behavior
What Googlebot does
Googlebot crawls pages automatically for the Google Search index, using a smartphone and a desktop crawler that share the same robots.txt token. It renders pages with an evergreen version of Chromium to execute JavaScript. Blocking it affects Google Search, Discover, Images, Video and News.
03 · Impact
Why Googlebot matters for your site
If you allow it
- Required to appear in Google Search
- Also feeds Discover, Google Images, Video and News
If you block it
- Only for pages you do not want in Google Search (use targeted Disallow rules)
- Spoofed Googlebot traffic should be blocked after verification
04 · Allow
How to allow Googlebot
robots.txt
User-agent: Googlebot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Googlebot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Googlebot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Googlebot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Googlebot
Googlebot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: Googlebot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Googlebot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Googlebot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Googlebot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Googlebot
The user agent published by the operator:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)07 · Verification
Is it really Googlebot?
Anyone can copy a user agent. Real Googlebot requests come from the IP ranges published at developers.google.com/static/crawling/ipranges/common-crawlers.json. A reverse DNS lookup of a real request resolves to googlebot.com, and that hostname resolves back to the same IP.
FAQ
Questions about Googlebot
How do I verify that a visitor is really Googlebot?
Run a reverse DNS lookup on the IP, check the hostname ends in googlebot.com, then confirm with a forward lookup; or match the IP against Google's published crawler IP ranges.
Can I block Googlebot Smartphone but not Googlebot Desktop?
No. Both use the same robots.txt token, so you cannot target one of them separately in robots.txt.
Does Googlebot render JavaScript?
Yes. Google renders pages with a headless, evergreen version of Chromium after crawling, when resources allow.
Last reviewed Oct 6, 2026