01 · Operator
Who operates DomainCrawler
- Company
- DomainCrawler
- Type
- Other bot
- Official docs
- domaincrawler.com
02 · Behavior
What DomainCrawler does
DomainCrawler builds databases of DNS, WHOIS, SSL certificate and web technology data. It says it has collected records since 2008. Clients use the data for business and security analysis.
03 · Impact
Why DomainCrawler matters for your site
If you allow it
- Keeps your domain's public data accurate in B2B datasets
If you block it
- Commercial data collection, no visitors
- No published verification method
04 · Allow
How to allow DomainCrawler
robots.txt
User-agent: DomainCrawler
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "DomainCrawler")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: DomainCrawler
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "DomainCrawler") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} DomainCrawler [NC]
# RewriteRule .* - [F,L]05 · Block
How to block DomainCrawler
Start with robots.txt. If DomainCrawler keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: DomainCrawler
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "DomainCrawler")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: DomainCrawler
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "DomainCrawler") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} DomainCrawler [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for DomainCrawler
The operator does not publish a full user agent string. Requests contain the token “DomainCrawler”.
07 · Verification
Is it really DomainCrawler?
DomainCrawler’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about DomainCrawler
What is DomainCrawler?
DomainCrawler is a Swedish B2B provider of structured domain data, such as DNS, WHOIS and web technology data.
Should I block DomainCrawler?
It is your call. It sends no visitors, but blocking it may leave its data about your domain outdated.
Last reviewed Oct 7, 2026