01 · Operator
Who operates Experibot
- Type
- Monitoring bot
- Official docs
- amirkr.wixsite.com
02 · Behavior
What Experibot does
It crawls websites for research. The collected data is not published and is used internally only. It waits at least 60 seconds between requests to the same host and re-reads robots.txt about every 12 hours.
03 · Impact
Why Experibot matters for your site
If you allow it
- Polite crawl rate of one request per minute per host
- Supports independent research
If you block it
- No traffic or visibility benefit
- Data use is not public
04 · Allow
How to allow Experibot
robots.txt
User-agent: experibot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "experibot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: experibot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "experibot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} experibot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Experibot
Experibot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: experibot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "experibot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: experibot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "experibot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} experibot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Experibot
The user agent published by the operator:
Experibot-v2 http://goo.gl/ZAr8wX07 · Verification
Is it really Experibot?
Experibot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Experibot
Does Experibot respect robots.txt?
Its documentation says it follows robots.txt disallow rules, including any rule for the token experibot. It refreshes robots.txt about every 12 hours.
Who runs Experibot?
Amir Krause runs it as a personal research project. Site owners can also ask by email to be added to its no-crawl list.
How do I block Experibot?
Add "User-agent: Experibot" followed by "Disallow: /" to your robots.txt, or block the user agent on your server.
Last reviewed Oct 8, 2026