01 · Operator
Who operates BDCbot
- Company
- BigDataCorp
- Type
- Other bot
- Official docs
- bigweb.bigdatacorp.com.br
02 · Behavior
What BDCbot does
BDCbot fetches public websites and stores the page HTML. The data enriches BigDataCorp's database, which it sells as data services to customers. Its user agent links to an FAQ on bigweb.bigdatacorp.com.br.
03 · Impact
Why BDCbot matters for your site
If you allow it
- Your company data may be more complete in BigDataCorp products
- Crawl is of public pages only
If you block it
- No traffic or visibility benefit
- Your content is resold as commercial data
- No published IP list for verification
04 · Allow
How to allow BDCbot
robots.txt
User-agent: BDCbot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "BDCbot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: BDCbot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "BDCbot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} BDCbot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block BDCbot
Start with robots.txt. If BDCbot keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: BDCbot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "BDCbot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: BDCbot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "BDCbot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} BDCbot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for BDCbot
The user agent published by the operator:
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.3607 · Verification
Is it really BDCbot?
BDCbot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about BDCbot
What is BDCbot?
BDCbot is BigDataCorp's crawler. It collects public web pages to enrich the company's commercial database.
How do I block BDCbot?
Add 'User-agent: BDCbot' and 'Disallow: /' to robots.txt. Its robots.txt behavior is not documented, so monitor your logs afterwards.
Is BDCbot harmful?
It is a data collection crawler, not an attack tool. It brings no visitors, so blocking it costs you nothing.
Last reviewed Oct 7, 2026