Blocked by robots.txt

Warning severity, in the Response codes tab

Why it matters

Blocked URLs cannot be crawled, so their content cannot be evaluated or indexed from the page itself.

How to fix it

If the page should be indexed, allow it in robots.txt. If not, this is expected.

How Crawlune finds it

Crawlune flags Blocked by robots.txt automatically on every crawl and lists the affected URLs under the Response codes tab, where you can filter, inspect each page and export the list to CSV. The free version crawls up to 550 URLs with every check enabled.