01 · Operator
Who operates AIWebIndex
- Company
- Aleksma Ai, Inc. (Lyrenth)
- Type
- AI search crawler
- Official docs
- lyrenth.com
02 · Behavior
What AIWebIndex does
AIWebIndex crawls public pages and converts them into clean documents that AI agents can access. Lyrenth says it is an indexer, not a model trainer. A separate AIWebIndex-Agent user agent handles on-demand fetches. Requests carry Web Bot Auth signatures.
03 · Impact
Why AIWebIndex matters for your site
If you allow it
- Follows robots.txt including Crawl-delay
- States it does not train models
- Publishes IP ranges and reverse DNS
If you block it
- Small operator with unclear traffic benefit
- Content is reused by third-party AI agents
04 · Allow
How to allow AIWebIndex
robots.txt
User-agent: AIWebIndex
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "AIWebIndex")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: AIWebIndex
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "AIWebIndex") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} AIWebIndex [NC]
# RewriteRule .* - [F,L]05 · Block
How to block AIWebIndex
AIWebIndex follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: AIWebIndex
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "AIWebIndex")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: AIWebIndex
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "AIWebIndex") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AIWebIndex [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for AIWebIndex
The user agent published by the operator:
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)07 · Verification
Is it really AIWebIndex?
Anyone can copy a user agent. Real AIWebIndex requests come from the IP ranges published at lyrenth.com/bot/ip-ranges.json. A reverse DNS lookup of a real request resolves to lyrenth.com, and that hostname resolves back to the same IP.
FAQ
Questions about AIWebIndex
Does AIWebIndex respect robots.txt?
Yes. Lyrenth says it honors Disallow and Crawl-delay per RFC 9309 and fetches at most once every 2 seconds per domain.
Does AIWebIndex train AI models?
Lyrenth describes AIWebIndex as an indexer, not a model trainer. It prepares pages for AI agents to read.
How can I verify AIWebIndex?
Check the IP against Lyrenth's published IP ranges and confirm forward-confirmed reverse DNS under lyrenth.com.
Last reviewed Oct 7, 2026