Error · Directives

Conflicting robots directives

Why it matters

The meta robots tag and the X-Robots-Tag header disagree; search engines apply the most restrictive.

How to fix it

Use one source of truth for robots directives.

How Crawlune finds it

Crawlune flags Conflicting robots directives automatically on every crawl and lists the affected URLs under theDirectives tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 500 URLs with every check enabled.