01 · Operator
Who operates SemrushBot
- Company
- Semrush
- Type
- SEO tool crawler
- Official docs
- semrush.com
02 · Behavior
What SemrushBot does
SemrushBot starts from a list of URLs and saves the links it finds for further crawling, building data for Semrush's tools. Semrush runs several other bots with their own robots.txt tokens, such as SiteAuditBot and SemrushBot-BA. It supports Crawl-delay and may take up to one hour or 100 requests to notice robots.txt changes.
03 · Impact
Why SemrushBot matters for your site
If you allow it
- Your site and backlinks appear correctly in Semrush data used by marketers
- Respects robots.txt and Crawl-delay
If you block it
- Competitors can use Semrush to analyze your site
- Crawl load with no direct traffic
- Cannot be blocked by IP because Semrush uses no consecutive IP blocks
04 · Allow
How to allow SemrushBot
robots.txt
User-agent: SemrushBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SemrushBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SemrushBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "SemrushBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block SemrushBot
SemrushBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: SemrushBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SemrushBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SemrushBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "SemrushBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for SemrushBot
The operator does not publish a full user agent string. Requests contain the token “SemrushBot”.
07 · Verification
Is it really SemrushBot?
SemrushBot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about SemrushBot
Does blocking SemrushBot affect Google rankings?
No. SemrushBot only collects data for Semrush; Google uses Googlebot.
Can I block SemrushBot by IP address?
Semrush advises against it because it does not use consecutive IP blocks; use robots.txt instead.
How long does SemrushBot take to notice robots.txt changes?
Semrush says it may take up to one hour or 100 requests for SemrushBot to discover changes.
Last reviewed Oct 6, 2026