01 · Operator
Who operates special_archiver
- Company
- Internet Archive (Archive-It)
- Type
- Other bot
- Official docs
- archive-it.org
02 · Behavior
What special_archiver does
It captures web pages for archive collections built by Archive-It partner institutions. Captured pages are stored for long-term preservation and later replay.
03 · Impact
Why special_archiver matters for your site
If you allow it
- Preserves your pages in web archives
- Used by libraries and research institutions
If you block it
- No direct traffic benefit
- Archived copies may persist after you change content
04 · Allow
How to allow special_archiver
robots.txt
User-agent: special_archiver
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "special_archiver")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: special_archiver
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "special_archiver") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} special_archiver [NC]
# RewriteRule .* - [F,L]05 · Block
How to block special_archiver
Start with robots.txt. If special_archiver keeps showing up in your logs, block it at your CDN or web server.
robots.txt
User-agent: special_archiver
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "special_archiver")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: special_archiver
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "special_archiver") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} special_archiver [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for special_archiver
The user agent published by the operator:
Mozilla/5.0 (compatible; special_archiver; Archive-It; +http://archive-it.org/files/site-owners-special.html)07 · Verification
Is it really special_archiver?
special_archiver’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about special_archiver
Who runs special_archiver?
It belongs to Archive-It, a web archiving service of the Internet Archive.
What does special_archiver do?
It captures pages for web archive collections built by Archive-It partner organizations.
Last reviewed Oct 8, 2026