Warning · Response codes

Blocked by robots.txt

Why it matters

Blocked URLs cannot be crawled, so their content cannot be evaluated or indexed from the page itself.

How to fix it

If the page should be indexed, allow it in robots.txt. If not, this is expected.

How Crawlune finds it

Crawlune flags Blocked by robots.txt automatically on every crawl and lists the affected URLs under theResponse codes tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 500 URLs with every check enabled.