Info · Directives

AI crawlers blocked in robots.txt

Why it matters

One or more AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) are fully disallowed, which removes the site from AI answers and training.

How to fix it

Decide deliberately which AI crawlers to allow; blocking them affects AI search visibility.

How Crawlune finds it

Crawlune flags AI crawlers blocked in robots.txt automatically on every crawl and lists the affected URLs under theDirectives tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 500 URLs with every check enabled.