01 · Operator
Who operates Diffbot
- Company
- Diffbot
- Type
- Other bot
- Official docs
- diffbot.com
02 · Behavior
What Diffbot does
Diffbot's Crawl product spiders a site for links. It hands those links to the Extract API, which converts each page into structured data. Results are compiled into a collection that customers can download or search.
03 · Impact
Why Diffbot matters for your site
If you allow it
- Your content may appear in structured datasets used by Diffbot customers
- Requests are tied to a known commercial operator
If you block it
- Crawls are run on behalf of Diffbot customers, not for search visibility
- Extracted data may be resold or used in other products
04 · Allow
How to allow Diffbot
robots.txt
User-agent: Diffbot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Diffbot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Diffbot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Diffbot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Diffbot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Diffbot
Start with robots.txt. If Diffbot keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: Diffbot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Diffbot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Diffbot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Diffbot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Diffbot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Diffbot
The user agent published by the operator:
Mozilla/5.0 (compatible; Diffbot/1.0; +https://diffbot.com)07 · Verification
Is it really Diffbot?
Diffbot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Diffbot
What is Diffbot?
Diffbot is a company that extracts structured data from web pages. Its Crawl product spiders sites and processes pages with its Extract API.
How do I block Diffbot?
Add a robots.txt group with User-agent: Diffbot and Disallow: /. Diffbot does not document whether every crawl honors robots.txt, so firewall rules are the stricter option.
Why does Diffbot crawl my site?
Diffbot customers start crawls of sites they want as structured data. The crawler then follows links and extracts page content.
Last reviewed Oct 8, 2026