01 · Operator
Who operates webXtrendBot
- Company
- webXtrend
- Type
- SEO tool crawler
- Official docs
- webxtrend.com
02 · Behavior
What webXtrendBot does
webXtrendBot fetches /robots.txt and then the start page of each domain, following redirects. webXtrend publishes DNS records, redirect targets, shared IPs and start-page properties. Customers use the data for market research, competitive analysis, IT security and brand protection. Each domain is revisited periodically.
03 · Impact
Why webXtrendBot matters for your site
If you allow it
- Only the start page is fetched, so load is low
- Requests can be verified by reverse DNS
- It obeys rules that name it explicitly
If you block it
- It ignores rules written for User-agent: *
- Your domain data is published on a third-party service
04 · Allow
How to allow webXtrendBot
robots.txt
User-agent: webXtrendBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "webXtrendBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: webXtrendBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "webXtrendBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} webXtrendBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block webXtrendBot
webXtrendBot does not always follow robots.txt. Add a server or CDN rule if you need a hard block.
robots.txt
User-agent: webXtrendBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "webXtrendBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: webXtrendBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "webXtrendBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} webXtrendBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for webXtrendBot
The user agent published by the operator:
Mozilla/5.0 (compatible; webXtrendBot/1.1; +https://webxtrend.com/crawler)07 · Verification
Is it really webXtrendBot?
Anyone can copy a user agent. A reverse DNS lookup of a real request resolves to .webxtrend.com, and that hostname resolves back to the same IP.
FAQ
Questions about webXtrendBot
What is webXtrendBot?
It is the crawler of webXtrend, a domain and DNS information service. It fetches robots.txt and the start page of domains.
Does webXtrendBot respect robots.txt?
Partly. It follows rules that name webXtrendBot, but webXtrend states that User-agent: * rules do not apply to it.
How can I verify webXtrendBot?
The reverse DNS name must end in .webxtrend.com and resolve forward to the same IP. No IP list is published.
Last updated Oct 19, 2026