AI training crawlers blocked in robots.txt

Info severity, in the AI search tab

Why it matters

One or more AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) are fully disallowed, which removes the site from AI answers and training.

How to fix it

Decide deliberately which AI crawlers to allow; blocking them affects AI search visibility.

How Crawlune finds it

Crawlune flags AI training crawlers blocked in robots.txt automatically on every crawl and lists the affected URLs under the AI search tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 550 URLs with every check enabled.