01 · Operator
Who operates peer39_crawler
- Company
- Peer39
- Type
- Other bot
- Official docs
- peer39.com
02 · Behavior
What peer39_crawler does
It analyzes page content to give advertisers contextual signals about relevance and brand suitability. Advertisers use this data to decide where to buy ad space. Peer39 states the crawler does not set cookies or track individual users.
03 · Impact
Why peer39_crawler matters for your site
If you allow it
- Pages classified as suitable can attract more programmatic ad demand
- No cookies or user tracking according to Peer39
If you block it
- No direct visibility or traffic benefit
- Uses server resources if you do not sell ads
04 · Allow
How to allow peer39_crawler
robots.txt
User-agent: peer39_crawler
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "peer39_crawler")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: peer39_crawler
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "peer39_crawler") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} peer39_crawler [NC]
# RewriteRule .* - [F,L]05 · Block
How to block peer39_crawler
peer39_crawler follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: peer39_crawler
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "peer39_crawler")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: peer39_crawler
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "peer39_crawler") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} peer39_crawler [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for peer39_crawler
| User agent | Share | Last seen | Status |
|---|---|---|---|
peer39_crawler/1.0 |
100% | Unverified |
07 · Verification
Is it really peer39_crawler?
peer39_crawler’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about peer39_crawler
What is peer39_crawler?
It is the crawler of Peer39, which classifies web page content for advertisers. The data supports contextual targeting and brand suitability.
Does peer39_crawler respect robots.txt?
Yes. Peer39 says the crawler reads robots.txt and follows its allow and disallow rules.
How do I block peer39_crawler?
Add "User-agent: peer39_crawler" followed by "Disallow: /" to your robots.txt file.
Last reviewed Oct 8, 2026