Domains Project Bot: What It Is & How to Block It

01 · Operator

Who operates Domains Project

Company: Domains Project (tb0hdan)
Type: Monitoring bot
Official docs: github.com

02 · Behavior

What Domains Project does

The crawler visits websites and DNS to find new hostnames. The project publishes a free list of about 1.7 billion domains on GitHub. A larger dataset is offered by subscription.

03 · Impact

Why Domains Project matters for your site

If you allow it

Open dataset used for internet research
Supports robots.txt opt-out

If you block it

No visitors or search benefit
robots.txt support is only partial

04 · Allow

How to allow Domains Project

robots.txt

User-agent: Domains Project
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Domains Project")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Domains Project
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Domains Project") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Domains Project [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block Domains Project

Domains Project does not always follow robots.txt. Add a server or CDN rule if you need a hard block.

robots.txt

User-agent: Domains Project
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Domains Project")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Domains Project
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "Domains Project") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Domains Project [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for Domains Project

The user agent published by the operator:

Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)

07 · Verification

Is it really Domains Project?

Domains Project’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

FAQ

Questions about Domains Project

What is the Domains Project crawler?

It is the crawler behind the Domains Project, an open dataset of internet domain names.

How do I block the Domains Project bot?

Add User-agent: Domains Project or User-agent: domainsproject.org with Disallow: /. The bot checks for both.

Does Domains Project respect robots.txt?

Partly. Its documentation says robots.txt support and rate limiting were added in version 1.0.7 and are partial.

Domains Project is Domains Project (tb0hdan)’s monitoring bot