Free tool

Robots.txt Generator

Decide which crawlers may read which parts of your site.

Build as many bot groups as you need: search engines, AI training bots, AI assistants or SEO tools, each with its own directories.

The bot list comes from the Log Hero crawler database.

All other crawlers

The fallback for every bot that isn't in one of your groups (User-agent: *).

Bot groups

Each group gets its own User-agent block with its own directories. Named bots ignore the * rules, so every group stands on its own.

or start from a template:

Sitemap & extras

Sitemap URLs One URL per line. Must be absolute. Crawl-delay Seconds, for all other crawlers. Google ignores it; Bing and Yandex respect it.

Know who is consuming your content

Your visitors are not the only audience for your website. Search engines crawl it. AI platforms consume it. Log Hero makes it visible and turns bot activity into data you can analyze.

Free during early access