01 · Operator
Who operates Bytespider
- Company
- ByteDance
- Type
- AI training crawler
02 · Behavior
What Bytespider does
Bytespider crawls pages automatically at high volume; Cloudflare reported in 2024 that it was the most active AI crawler on its network. It is reportedly used to gather training data for ByteDance's LLMs, including those behind Doubao. It is not triggered by users, and ByteDance publishes no official IP list or verification method.
03 · Impact
Why Bytespider matters for your site
If you allow it
- Content may reach ByteDance products
- No other documented benefit to site owners
If you block it
- Reported to ignore robots.txt
- High crawl volume
- No published IP list or verification method
04 · Allow
How to allow Bytespider
robots.txt
User-agent: Bytespider
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Bytespider")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Bytespider
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "Bytespider") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
# RewriteRule .* - [F,L]05 · Block
How to block Bytespider
Bytespider has been reported to ignore robots.txt. Block it at your CDN or web server as well.
robots.txt
# Bytespider has been reported to ignore robots.txt.
# Add a Cloudflare, nginx or Apache rule as well.
User-agent: Bytespider
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "Bytespider")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: Bytespider
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "Bytespider") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for Bytespider
The operator does not publish a full user agent string. Requests contain the token “Bytespider”.
07 · Verification
Is it really Bytespider?
Bytespider’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about Bytespider
Who operates Bytespider?
ByteDance, the Chinese company that owns TikTok.
Does Bytespider respect robots.txt?
Reporting by Fortune in 2024, citing Kasada and Dark Visitors, said Bytespider does not respect robots.txt, so a server or firewall block is more reliable.
Why is Bytespider crawling my site so much?
Cloudflare reported in 2024 that Bytespider had the highest request volume of the AI crawlers it tracked, reaching about 40% of sites it protects.
Last reviewed Oct 6, 2026