01 · Operator
Who operates Observer
- Company
- Observer (obsrvr.net)
- Type
- Monitoring bot
- Official docs
- obsrvr.net
02 · Behavior
What Observer does
Observer checks sites for compromised external resources and maliciously re-registered expired domains. It also flags performance bottlenecks. It reads robots.txt before crawling and spaces out its requests. Its IPs resolve to crawl.obsrvr.net.
03 · Impact
Why Observer matters for your site
If you allow it
- Respects robots.txt and throttles requests
- Can flag compromised third-party resources
If you block it
- No search traffic
- Findings go to the service, not necessarily to you
04 · Allow
How to allow Observer
robots.txt
User-agent: Observer
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Observer")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Observer
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Observer") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Observer [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Observer
Observer follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: Observer
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Observer")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Observer
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Observer") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Observer [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Observer
| User agent | Share | Last seen | Status |
|---|---|---|---|
Observer/1.0 (+https://obsrvr.net/about; crawler@obsrvr.net) |
100% | Unverified |
07 · Verification
Is it really Observer?
Anyone can copy a user agent. A reverse DNS lookup of a real request resolves to crawl.obsrvr.net, and that hostname resolves back to the same IP.
FAQ
Questions about Observer
What is the Observer crawler?
It is the crawler of obsrvr.net, a service that detects web security and third-party risks.
Does Observer respect robots.txt?
Yes. Observer says it checks and respects robots.txt before any crawling.
How do I verify Observer?
Run a reverse DNS lookup. Genuine requests resolve to crawl.obsrvr.net.
Last reviewed Oct 8, 2026