CrawluneLog file analysis
A crawl shows what your site offers; the server logs show what search engines actually take. Crawlune Pro reads the access logs and joins every request to the crawled URLs, so you can see which pages Googlebot fetches, which it ignores, where it wastes crawl budget, and which URLs it keeps requesting that no longer exist in your site structure.
1. Get the logs
- Apache / Nginx: the access log in combined or common format, usually /var/log/apache2/access.log or /var/log/nginx/access.log, with the rotated .gz files for history. A vhost prefix (example.com:443 1.2.3.4 …) is fine.
- IIS: the W3C extended log files from %SystemDrive%\inetpub\logs\LogFiles; the #Fields: header tells Crawlune the column order. Include the user agent and client IP fields.
- Cloudflare, Fastly and friends: JSON lines exports such as Cloudflare Logpush (HTTP requests data set). Crawlune reads the usual key names: path or URI, status, user agent, client IP and timestamp.
- Two to four weeks of logs give a stable picture; longer windows are fine, the import streams the files and never loads them whole.
2. Import
- Crawl the site as usual and let it finish.
- Press the Log files button in the toolbar and choose the files. Leave Verify Googlebot and Bingbot IPs on: Crawlune downloads Google's and Bing's published crawler ranges once a week and counts requests from other IPs as fake.
- The dialog sums the requests per crawler, draws bot hits per day, lists the response codes and the crawlers seen, and shows the URLs verified bots requested that the crawl did not reach.
From a script: crawlune crawl https://example.com --logs access.log --logs access.log.1.gz, or POST /v1/logs on the local API, or the import_logs MCP tool.
3. What you get
| Where | What |
|---|---|
| Log files tab | Per URL: total hits, Googlebot, Bingbot, AI bots, other bots, humans, errors, the status of the last request and when it happened. Filters for pages Googlebot never fetched, bots receiving errors, and more. |
| Detail pane | A Log files row on every page with the same counts. |
| Checks | Indexable page Googlebot never fetched · Bots receive errors · Googlebot spends crawl budget on a non-indexable page · Bots keep hitting a redirect · URLs bots fetch that the crawl did not reach · Fake Googlebot or Bingbot traffic. They feed the health score, the action plan and the PDF report like every other check. |
| Orphan list | Paths with a 200 response to verified search or AI crawlers that no crawled page links to. Copy them into the list crawl to audit them, then link, redirect or retire. |
How requests are classified
- Googlebot: Googlebot, Google-InspectionTool, AdsBot, Storebot, GoogleOther, APIs-Google and Mediapartners. Bingbot: bingbot, BingPreview, msnbot, adidxbot. Both are verified by IP when verification is on.
- AI bots: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, CCBot, Bytespider, Google-Extended, Applebot-Extended, Meta and Amazon crawlers, DuckAssistBot.
- Other bots: everything else that names itself a bot, crawler or spider, plus SEO tools and command-line clients. Humans: the rest.
- Only GET and HEAD requests count. Paths are matched to crawled URLs ignoring the trailing slash; query strings are kept.
Logs are read on your machine and never uploaded. The only network request is the weekly download of the crawler IP ranges, which you can turn off.