Opportunity · Directives

No robots.txt

Why it matters

Without robots.txt every crawler must guess, and you cannot declare sitemaps or block crawl traps.

How to fix it

Publish a robots.txt, even a permissive one, and reference your sitemap in it.

How Crawlune finds it

Crawlune flags No robots.txt automatically on every crawl and lists the affected URLs under theDirectives tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 500 URLs with every check enabled.