DesearchBot User Agent: What It Is & How to Block It

01 · Operator

Who operates DesearchBot

Company: Desearch
Type: AI search crawler
Official docs: desearch.ai

02 · Behavior

What DesearchBot does

DesearchBot fetches public web pages for the Desearch search service. It sends at most one request per second per host. Every request is signed with Ed25519 per RFC 9421 (HTTP message signatures).

03 · Impact

Why DesearchBot matters for your site

If you allow it

Can make your pages findable in Desearch
Follows robots.txt and Crawl-delay
Requests are cryptographically signed

If you block it

Small search service with limited traffic
Content may be served to AI agents

04 · Allow

How to allow DesearchBot

robots.txt

User-agent: DesearchBot
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "DesearchBot")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: DesearchBot
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "DesearchBot") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} DesearchBot [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block DesearchBot

DesearchBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.

robots.txt

User-agent: DesearchBot
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "DesearchBot")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: DesearchBot
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "DesearchBot") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} DesearchBot [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for DesearchBot

The user agent published by the operator:

Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler)

07 · Verification

Is it really DesearchBot?

DesearchBot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

FAQ

Questions about DesearchBot

Does DesearchBot respect robots.txt?

Yes. Desearch says it observes Disallow, Allow and Crawl-delay for its token and falls back to the * group.

How can I verify DesearchBot?

Desearch publishes no IP list. Verify the request signature using the key directory named in the Signature-Agent header.

DesearchBot is Desearch’s AI search crawler