01 · Operator
Who operates SiteAuditBot
- Company
- Semrush
- Type
- SEO tool crawler
- Official docs
- semrush.com
02 · Behavior
What SiteAuditBot does
It crawls a website on request of a Semrush user to find SEO and technical issues. It can run as a mobile or desktop crawler and can execute JavaScript if enabled. Site owners can let it bypass robots.txt disallow rules by uploading a verification file.
03 · Impact
Why SiteAuditBot matters for your site
If you allow it
- Your own team or agency may use Semrush Site Audit
- It follows robots.txt by default
If you block it
- Third parties can audit your site without your consent
- It brings no direct traffic
- Default speed is about one page per second
04 · Allow
How to allow SiteAuditBot
robots.txt
User-agent: SiteAuditBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SiteAuditBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SiteAuditBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "SiteAuditBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} SiteAuditBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block SiteAuditBot
SiteAuditBot does not always follow robots.txt. Add a server or CDN rule if you need a hard block.
robots.txt
User-agent: SiteAuditBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SiteAuditBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SiteAuditBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "SiteAuditBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} SiteAuditBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for SiteAuditBot
The user agent published by the operator:
Mozilla/5.0 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)07 · Verification
Is it really SiteAuditBot?
SiteAuditBot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about SiteAuditBot
How do I block SiteAuditBot?
Add User-agent: SiteAuditBot with Disallow: / to robots.txt. Semrush lists this token on its bot page.
Can I limit SiteAuditBot's crawl speed?
Yes. Add Crawl-delay under User-agent: SiteAuditBot. Semrush caps the applied delay at 30 seconds, and only if the audit is set to respect robots.txt.
Why does SiteAuditBot look like an iPhone?
Site Audit can crawl as a mobile or desktop bot. The mobile setting is the default and audits a site as Google's mobile crawler would.
Last reviewed Oct 7, 2026