IRLbot User Agent: Texas A&M Research Crawler

01 · Operator

Who operates IRLbot

Company: Texas A&M University, Internet Research Lab (IRL)
Type: Monitoring bot
Official docs: irl.cs.tamu.edu

02 · Behavior

What IRLbot does

IRLbot downloads random text-only pages and follows links to other sites. The lab uses it to research algorithms for mapping Internet topology. Downloaded text is not distributed publicly or used for non-research purposes.

03 · Impact

Why IRLbot matters for your site

If you allow it

Supports academic web research
Low default rate of one page per 40 seconds per site

If you block it

No search or traffic benefit to site owners
Bandwidth use on large sites

04 · Allow

How to allow IRLbot

robots.txt

User-agent: IRLbot
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "IRLbot")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: IRLbot
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "IRLbot") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} IRLbot [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block IRLbot

IRLbot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.

robots.txt

User-agent: IRLbot
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "IRLbot")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: IRLbot
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "IRLbot") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} IRLbot [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for IRLbot

The user agent published by the operator:

IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

07 · Verification

Is it really IRLbot?

IRLbot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

FAQ

Questions about IRLbot

Who runs IRLbot?

IRLbot is a research project of the Internet Research Lab at Texas A&M University. Questions go to Dr. Dmitri Loguinov.

How do I block IRLbot?

Add "User-agent: IRLbot" and "Disallow: /" to robots.txt. It also honors Crawl-delay and may cache robots.txt for 1 to 7 days.

How can I verify real IRLbot traffic?

The lab says IRLbot normally runs from 128.194.*.*. It treats other IPs using the IRLbot user agent as impersonators.

IRLbot is Texas A&M University, Internet Research Lab (IRL)’s monitoring bot