AI training crawlers allowed

Info severity, in the AI search tab

Why it matters

No AI training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider and others) is disallowed, so page content may be used to train models. This is a policy choice, not an error.

How to fix it

If you want to opt out of training while staying visible in AI search, add Disallow rules for the training agents only.

How Crawlune finds it

Crawlune flags AI training crawlers allowed automatically on every crawl and lists the affected URLs under the AI search tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 550 URLs with every check enabled.