Log in
Sign up with Google
Free tool

Robots.txt Generator

Decide which crawlers may read which parts of your site.

Build as many bot groups as you need: search engines, AI training bots, AI assistants or SEO tools, each with its own directories.

The bot list comes from the Log Hero crawler database.

New to robots.txt? See how it works
01

Default rules

The fallback for every bot that isn't in one of your groups (User-agent: *).

02

Bot groups

Each group gets its own User-agent block with its own directories. Named bots ignore the * rules, so every group stands on its own.

or start from a template:
03

Sitemap & extras

How it works

How to use the Robots.txt Generator

Four steps from a blank file to a robots.txt that tells every crawler exactly what it may read.

  1. 01

    Set the default for all crawlers

    Choose whether bots you don't name may crawl everything (Allow all) or nothing (Block all). Then add the directories that should be excluded, such as /wp-admin/ or /checkout/.

  2. 02

    Create bot groups

    Add a group, give it a name and pick bots from the dropdown. Search by bot or company, or add a whole category like AI training crawlers at once. Templates give you a head start.

  3. 03

    Set the rules per group

    Block the group completely, allow it completely, or define custom directories with Allow and Disallow. “Copy default directories” takes over the rules from step 01 as a starting point.

  4. 04

    Download and upload

    Add your sitemap, check the live preview and the hints next to it, then download the file. Upload it to the root of your domain so it's reachable at yourdomain.com/robots.txt.

Good to know

Named bots ignore the * rules

A crawler that finds its own user-agent group follows only that group. That's why every group has its own directories, and why each bot can only be in one group.

Wildcards keep rules short

* matches any sequence of characters, $ marks the end of a URL. /*.pdf$ blocks every PDF, /*?*utm_ every URL with tracking parameters.

Blocking AI training ≠ blocking AI search

Training crawlers like GPTBot collect data for models. Assistants like ChatGPT-User or PerplexityBot fetch pages to answer questions and link back to you. Decide for each group separately.

A request, not a lock

Well-behaved crawlers follow robots.txt, others don't. Your server logs show which bots really visit and whether they respect your rules. That's exactly what Log Hero makes visible.

Know who is consuming your content

Your visitors are not the only audience for your website. Search engines crawl it. AI platforms consume it. Log Hero makes it visible and turns bot activity into data you can analyze.

Sign up with Google Read the documentation

Free during early access