Butterfly Bot (Topsy): Is It Still Active? How to Block It

01 · Operator

Who operates Butterfly

Company: Topsy Labs (defunct)
Type: Other bot

02 · Behavior

What Butterfly does

Its user agent pointed to labs.topsy.com. Topsy offered real-time search and analytics over Twitter content. The crawler is no longer operated, so current requests with this name are likely spoofed.

03 · Impact

Why Butterfly matters for your site

If you allow it

Historically supported Topsy's social search

If you block it

Operator shut down in 2015
Current traffic is likely fake

04 · Allow

How to allow Butterfly

robots.txt

User-agent: Butterfly
Allow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Butterfly")
Action:     Skip › All Super Bot Fight Mode rules

# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Butterfly
Allow: /

nginx

# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Butterfly") { return 403; }

Apache

# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Butterfly [NC]
# RewriteRule .* - [F,L]

05 · Block

How to block Butterfly

Start with robots.txt. If Butterfly keeps showing up in your logs, block it at your CDN or web server.

robots.txt

User-agent: Butterfly
Disallow: /

Cloudflare

# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Butterfly")
Action:     Block

WordPress

# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Butterfly
Disallow: /

nginx

# In your server { } block:
if ($http_user_agent ~* "Butterfly") {
    return 403;
}

Apache

# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Butterfly [NC]
RewriteRule .* - [F,L]
</IfModule>

06 · User agents

User agents we see for Butterfly

The user agent published by the operator:

Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8

07 · Verification

Is it really Butterfly?

Butterfly’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.

FAQ

Questions about Butterfly

Is the Butterfly crawler still active?

No. Topsy, its operator, shut down in December 2015 after Apple acquired it.

Should I block Butterfly?

Yes, it is safe to block. Any traffic using this user agent today does not come from Topsy.

What was Topsy?

Topsy was a San Francisco company that offered real-time search and analytics for Twitter content.

Butterfly was Topsy Labs (defunct)’s other bot