AdsTxtCrawler User Agent: What It Is & How to Block It

01 · Operator

Who operates AdsTxtCrawler

Company: IAB Technology Laboratory
Type: Other bot
Official docs: github.com

02 · Behavior

What AdsTxtCrawler does

It fetches the ads.txt file from a list of domains and stores the records in a SQLite database. It requests hostname/ads.txt only. Because it is open source, anyone can run it.

03 · Impact

Why AdsTxtCrawler matters for your site

If you allow it

Only reads ads.txt
Helps ad buyers verify authorized sellers

If you block it

Anyone can run it
Not useful if you do not sell ads

04 · Allow

How to allow AdsTxtCrawler

robots.txt

User-agent: AdsTxtCrawler
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "AdsTxtCrawler")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: AdsTxtCrawler
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "AdsTxtCrawler") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} AdsTxtCrawler [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block AdsTxtCrawler

Start with robots.txt. If AdsTxtCrawler keeps showing up in your logs, block it at your CDN or web server.

robots.txt

User-agent: AdsTxtCrawler
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "AdsTxtCrawler")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: AdsTxtCrawler
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "AdsTxtCrawler") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AdsTxtCrawler [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for AdsTxtCrawler

The user agent published by the operator:

AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler

07 · Verification

Is it really AdsTxtCrawler?

AdsTxtCrawler’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

FAQ

Questions about AdsTxtCrawler

What does AdsTxtCrawler fetch?

It fetches the ads.txt file of each target domain and stores the entries in a database.

Who runs AdsTxtCrawler?

The IAB publishes it as open-source reference code. Any ad-tech company or individual can run it.

AdsTxtCrawler is IAB Technology Laboratory’s other bot