01 · Operator
Who operates bnf.fr_bot
02 · Behavior
What bnf.fr_bot does
The BnF archives the French web as required by the French Heritage Code. The robot uses the Heritrix crawler and applies delays between requests. It may collect files covered by robots.txt when needed to rebuild a site's original form. Heritrix cannot process all JavaScript, so it may request false URLs.
03 · Impact
Why bnf.fr_bot matters for your site
If you allow it
- Preserves your site in France's national web archive
- Legal deposit is a legal mandate for French sites
- Polite crawl rate
If you block it
- It may fetch files disallowed in robots.txt
- Can request invalid URLs from JavaScript
04 · Allow
How to allow bnf.fr_bot
robots.txt
User-agent: bnf.fr_bot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "bnf.fr_bot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: bnf.fr_bot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "bnf\.fr_bot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} bnf\.fr_bot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block bnf.fr_bot
bnf.fr_bot does not always follow robots.txt. Add a server or CDN rule if you need a hard block.
robots.txt
User-agent: bnf.fr_bot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "bnf.fr_bot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: bnf.fr_bot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "bnf\.fr_bot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} bnf\.fr_bot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for bnf.fr_bot
The user agent published by the operator:
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)07 · Verification
Is it really bnf.fr_bot?
bnf.fr_bot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about bnf.fr_bot
Why is bnf.fr_bot crawling my site?
The BnF archives French websites under the legal deposit of the internet. Your site was selected for its collections.
Does bnf.fr_bot respect robots.txt?
Partly. The BnF is legally allowed to ignore robots.txt and may collect blocked files to reconstruct a site.
Can I stop the BnF from archiving my site?
Legal deposit is set by French law (Heritage Code art. L131-1 ff.). Contact the BnF if the crawl causes technical problems.
Last reviewed Oct 7, 2026