AI search bots blocked in robots.txt

Warning severity, in the AI search tab

Why it matters

robots.txt fully disallows crawlers that fetch pages to answer questions and cite sources (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and similar). The site cannot appear or be cited in those AI answers.

How to fix it

Allow the AI search and answer agents you want citations from; keep separate rules for training crawlers if you want to restrict those.

How Crawlune finds it

Crawlune flags AI search bots blocked in robots.txt automatically on every crawl and lists the affected URLs under the AI search tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 550 URLs with every check enabled.