01 · Operator
Who operates netEstate NE Crawler
- Company
- netEstate GmbH
- Type
- Other bot
02 · Behavior
What netEstate NE Crawler does
netEstate runs its own crawlers to extract structured data from websites. Its products include crawlers for imprint data, such as company names and addresses, and for job postings. The user agent links to website-datenbank.de, a website database.
03 · Impact
Why netEstate NE Crawler matters for your site
If you allow it
- Identifies itself with a contact URL
- Your site may be listed in a German website database
If you block it
- No direct search traffic
- Your contact and company data may be extracted into commercial products
04 · Allow
How to allow netEstate NE Crawler
robots.txt
User-agent: netEstate NE Crawler
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "netEstate NE Crawler")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: netEstate NE Crawler
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "netEstate NE Crawler") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} netEstate NE Crawler [NC]
# RewriteRule .* - [F,L]05 · Block
How to block netEstate NE Crawler
Start with robots.txt. If netEstate NE Crawler keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: netEstate NE Crawler
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "netEstate NE Crawler")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: netEstate NE Crawler
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "netEstate NE Crawler") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} netEstate NE Crawler [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for netEstate NE Crawler
The user agent published by the operator:
netEstate NE Crawler (+http://www.website-datenbank.de/)07 · Verification
Is it really netEstate NE Crawler?
netEstate NE Crawler’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about netEstate NE Crawler
Who operates netEstate NE Crawler?
Matomo's bot database attributes it to netEstate GmbH, a German information extraction company.
What data does netEstate collect?
netEstate builds products that extract imprint data and job postings from websites.
How do I block netEstate NE Crawler?
Block the user agent at your server or firewall. Its robots.txt behavior is not documented.
Last reviewed Oct 8, 2026