01 · Operator
Who operates BotSeer
- Type
- Monitoring bot
- Official docs
- en.wikipedia.org
02 · Behavior
What BotSeer does
BotSeer collected and analyzed robots.txt files to study how sites treat web robots. It indexed 2.2 million robots.txt files from 13.2 million websites. Yang Sun, Isaac G. Councill, Ziming Zhuang and C. Lee Giles built it. The project is no longer active.
03 · Impact
Why BotSeer matters for your site
If you allow it
- Historical research interest only
If you block it
- The project ended around 2010
- Current traffic with this name is likely not from the project
04 · Allow
How to allow BotSeer
robots.txt
User-agent: BotSeer
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "BotSeer")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: BotSeer
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "BotSeer") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} BotSeer [NC]
# RewriteRule .* - [F,L]05 · Block
How to block BotSeer
Start with robots.txt. If BotSeer keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: BotSeer
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "BotSeer")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: BotSeer
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "BotSeer") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} BotSeer [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for BotSeer
The operator does not publish a full user agent string. Requests contain the token “BotSeer”.
07 · Verification
Is it really BotSeer?
BotSeer’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about BotSeer
Is BotSeer still active?
No. BotSeer was in operation from about 2007 to 2010.
What did BotSeer do?
It was a search tool for research on web robots. It indexed and analyzed millions of robots.txt files.
Should I block BotSeer?
Yes, blocking it has no downside. The research project is no longer running.
Last reviewed Oct 7, 2026