Skip to content
Crawlune

Changelog

All notable changes to Crawlune. The format follows Keep a Changelog; versions follow SemVer.

Unreleased

  • One-click activation: crawlune://activate/<key> links from the account page and the licence emails activate a trial or purchased key without copy and paste (single-instance on Windows and Linux so a second launch reaches the running app).
  • “Start free trial” in the upgrade dialog and the Licence panel opens the account page with the trial pre-selected; the upgrade dialog also links the new Pro Monthly plan ($19 a month).
  • First-run welcome dialog links the Terms of service and Privacy policy.

0.9.0 — 2026-10-18

  • Pro: screenshot capture. Settings → Rendering can save a 1366×900 screenshot of every rendered page; it shows in the detail pane’s Screenshot tab, is kept with the saved crawl, and the start page’s shot opens the PDF report. CLI --screenshots.
  • Extraction templates. Settings → Extraction has “Add from a template”: product price, SKU and availability; article author and dates; Open Graph and Twitter card tags; schema.org types; breadcrumbs; GA4 and GTM ids; headings; contact details.
  • Pro: issue status workflow. Every finding in the detail pane gets a status (open, in progress, fixed, false positive) and a note, kept per site across crawls. A finding marked fixed that comes back in a later crawl shows as regressed. The sidebar counts resolved findings per check, “hide resolved” takes them out of the affected-URL views, and the check’s context menu marks all its findings at once. CI mode no longer fails on findings marked false positive.
  • Pro: AI provider integration. Settings → Integrations takes an OpenAI, Anthropic or Gemini key, or a local Ollama, and the new AI button runs three jobs on the crawl: semantic analysis (embeddings find pages that say the same thing in different words and pages off the site’s topic; checks “Semantic duplicate” and “Off-topic for this site”), title and meta description suggestions for flagged pages with copy and CSV export, and proofreading in any language (findings per page in the detail pane; check “Spelling or grammar mistakes”). Page text goes only to the provider you chose.
  • Pro: log file analysis. The new Log files button (and crawl --logs) reads Apache, Nginx, IIS and JSON-lines access logs, plain or gzip, and joins them to the crawl: hits per URL by Googlebot, Bingbot, AI crawlers, other bots and humans, with Googlebot and Bingbot verified against the published IP ranges. The dialog shows bot hits per day, crawlers seen, response codes and the URLs bots fetch that the crawl did not reach. New Log files tab and six checks: indexable pages Googlebot never fetched, bots receiving errors, budget spent on non-indexable pages, bots hitting redirects, orphan URLs, fake Googlebot/Bingbot traffic. Also POST /v1/logs and the import_logs MCP tool.

0.8.0 — 2026-10-18

  • Pro: local REST API. Settings → Integrations → “Local API and MCP server” starts an HTTP listener on 127.0.0.1 (port 7420 by default) protected by a token the app generates: start, pause, stop and poll crawls, read issues, pages, page details and HTML, the action plan, the checks catalogue and the library from scripts. Docs at crawlune.com/docs/api/. crawlune serve runs the same API without a window.
  • Pro: MCP server. crawlune mcp lets Claude Desktop, Claude Code, Cursor and other assistants run crawls and query results through 15 tools (start_crawl, wait_for_crawl, list_issues, issue_pages, page_details, action_plan, explain_check…); the settings panel shows the client config to paste.

0.7.0 — 2026-10-17

  • Pro: Looker Studio data set. Export → Google Sheets → “Looker Studio data set” keeps one spreadsheet per site in your Drive with a Crawls sheet (one row per crawl), an Issues sheet (check by crawl) and a Pages sheet (the latest crawl with Search Console, Analytics and PageSpeed columns). Build a Looker Studio report on it once and it stays current; the guide is at crawlune.com/docs/looker-studio/.
  • Pro: notifications and alerts. Settings → Integrations takes a Slack incoming webhook, a generic webhook URL and email addresses; each schedule can notify after every run or only when a rule fires: health dropped by N points, errors grew by N, chosen checks have new findings, or the crawl failed. Slack and webhooks are posted straight from the app; email goes through crawlune.com. “Send a test” shows what arrives.
  • Pro: proxy support. Settings → Robots & agent takes an http, https or socks5 proxy URL, with credentials in the URL, used for every request of the crawl; the renderer uses the proxy host too.
  • Pro: multi-domain crawls. Settings → Scope → “More sites crawled as one” seeds each extra site from its home page and treats its host as internal, so a shop on another domain or a country site is audited together with the main site.
  • Pro: library retention. The saved-crawls dialog can cap crawls per site and by age; the limits apply after every save.
  • Pro: redirect audit. The list dialog’s third tab checks old URLs after a migration: every hop is followed by hand, and chains, loops, temporary redirects, non-200 endings and wrong destinations (when you give the expected URL) are flagged, with a CSV export.
  • Staging versus production: comparing two saved crawls of different hosts now matches URLs by path, so a staging crawl lines up against the live site.

0.6.0 — 2026-10-15

  • Trends: the Visualise dialog’s third tab charts site health, errors, warnings, opportunities and URL counts across every saved crawl of a site, with a per-check table showing first, latest and change plus a sparkline. Per-check history is recorded for crawls saved from this version on.
  • Visualise the crawl (toolbar button or palette): a directory tree of the site with pages, issues and average link score per folder, click a folder to filter the table; and a crawl map, a force-directed graph of internal links sized by link score and coloured by status, click a page to select it. Large sites show the best-linked pages first.
  • Pro: email the report. Export → Email report… sends the PDF, CSV or Excel workbook to any recipients: the files go through crawlune.com, which emails download links valid for 30 days, with your account address as the sender to reply to. Scheduled crawls can do the same after every run, so a monthly client audit arrives in the client’s inbox without you opening the app.
  • Pro: accessibility audit. With JavaScript rendering on, Settings → Rendering can run axe-core (WCAG 2.2 A and AA plus best practices) in every rendered page. The new Accessibility tab counts violations by impact, the detail pane lists each rule with the affected element count and a link to the fix, and seven checks flag critical, serious and moderate violations, colour contrast, unnamed links, buttons and fields, a missing page language, and missing landmarks. CLI: --a11y.
  • Pro: PageSpeed Insights and Core Web Vitals. With your own free PageSpeed API key (Settings → Integrations), Crawlune tests the best-linked indexable pages after a crawl, or on demand: Lighthouse performance score, LCP, CLS and total blocking time, plus real-user field data (LCP, INP, CLS and the pass/fail assessment) where Google has it. A new Performance tab and six checks: score under 50, score 50–89, LCP over 4 s, CLS over 0.25, INP over 500 ms, and Core Web Vitals not passing.
  • Pro: Google Analytics 4. The same Google sign-in now joins sessions, views, engagement rate and key events per URL to the crawl, in a new Analytics tab, the CSV export and the detail pane, on demand or after every crawl. Five checks: indexable pages with no sessions, low engagement, traffic landing on redirecting or non-indexable pages, pages with traffic but no search impressions, and URLs with sessions the crawl never reached.

0.5.0 — 2026-10-10

  • Pro: Google Search Console. Connect your Google account once (Settings → Integrations) and Crawlune joins clicks, impressions, CTR and average position to every crawled URL, in a new Search Console tab, the CSV export and the detail pane. Fetch for a finished crawl or after every crawl, scheduled ones included. Five new checks: indexable pages with no impressions, page-one rankings with a low CTR, pages just off page one, non-indexable pages still getting clicks, and URLs Google shows that the crawl never reached.
  • HTML validation: three new checks in the Content tab read the raw markup rather than the repaired DOM. Content before the doctype (PHP output, BOMs, template whitespace), elements that do not belong in the head (a stray div, img or early body, which makes browsers close the head so later meta and canonical tags land in the body) and duplicate html, head or body tags. The detail pane lists what was found; an HTML problems column joins the Content tab and the CSV export.
  • PDF report: the health score no longer overlaps the “Executive summary” heading.

0.4.0 — 2026-10-09

  • Pro: sitemap mode. Crawl exactly what a site’s XML sitemaps list (discovered through robots.txt and /sitemap.xml, or named explicitly; sitemap indexes are followed) to check that every listed URL is live, indexable and canonical. In the list dialog and on the CLI (--sitemap).
  • List mode: load URLs from a text, CSV or XML file (first column, or <loc> entries); CLI --list <file>. Pro crawls lists of any size.
  • Pro: Excel export. Export → Excel writes the current view, all URLs, or every issue on its own sheet (with a summary sheet), with bold frozen headers, filters and sized columns. Scheduled crawls can write the workbook too.
  • Pro: Google Sheets export. Export → Google Sheets creates a spreadsheet in your Google Drive from the current view or all URLs and opens it. Sign in once from Settings → Report; Crawlune only gets access to files it creates.
  • Crash reporting, opt in: if Crawlune crashes, the next launch offers to send a report (app version, OS, error and stack trace; never URLs, crawl data or keys), with an optional note and contact. “Always send” is available.
  • In-app updates: the Update button now downloads the signed update and installs it; one click to restart. Linux .deb installs still update through the download page.
  • Command line: the app binary doubles as a CLI. crawlune crawl <url> runs headless with a summary and optional JSON, CSV, PDF and library save; crawlune ci <url> (Pro) compares with a baseline, writes JUnit and SARIF, and exits non-zero on new findings at or above a severity, for pipelines. On macOS: Crawlune.app/Contents/MacOS/crawlune.
  • Pro: scheduled crawls. Settings → Schedules runs a site daily, weekly or monthly while Crawlune is open, with the current settings; results land in saved crawls and can be exported as a CSV and a PDF report to a folder. Run now, pause and edit.
  • Pro: configuration profiles. Save all settings under a name from the top of the settings dialog and switch between them.
  • Pro: developer tickets. From any issue (sidebar right-click, a URL’s issue, or the action plan) generate a ticket as Markdown, Jira wiki markup or plain text: why it matters, how to fix, affected URLs with key pages first, acceptance criteria and a link to the check’s page. Copy or save it.
  • AI search tab: a new check category for visibility in AI answers. robots.txt is read per bot, separating AI search and answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and others) from training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider and others); llms.txt is checked for a title, summary and links; pages are checked for a main or article landmark, content buried in markup, and noai directives.
  • Internal PageRank: every internal page gets a link score (0–100) from the flow of followed internal links, with redirects passing equity on. Shown as a column in the Internal and Links tabs; the action plan’s “on key pages” weighting now uses it.
  • Pro: segments. Label URLs by regex under Settings → Scope; a Segment column, per-segment filters in the Internal and Links tabs, and a per-segment table in the PDF report.
  • Pro: custom extraction and search. Define CSS selectors (text, inner HTML or an attribute) and regexes (first capture group) to pull values from every page, and searches (plain or regex, source or visible text) counted per page. Results appear as columns in the new Custom tab, with has/missing and match/no-match filters, in CSV exports, and are taken from the rendered DOM when rendering is on. Settings → Extraction.
  • Pro: ignore rules. Right-click an issue in the sidebar, or use Ignore… on a URL’s issue, to hide an accepted finding for this site or everywhere, optionally for a URL pattern. Hidden findings leave the counts, health score, action plan and report; rules persist across crawls and are managed under Settings → Ignore rules.
  • Hreflang validation: conflicting entries for one language, relative hrefs, targets that are not indexable, and a page language that disagrees with its own hreflang.
  • Pagination: rel=next/prev that points to itself, series that do not link back, and paginated URLs without annotations.
  • Near-duplicate content: pages whose text is almost identical (not just byte-for-byte) are flagged, with the similar URLs and a similarity percentage in the detail pane.
  • Structured data validation: JSON-LD blocks are parsed and checked for valid JSON, @context, @type and the properties required for rich results (Article, Product, Offer, Organization, WebSite, BreadcrumbList, FAQPage, LocalBusiness, Event, Recipe, VideoObject, JobPosting and more); microdata types are listed. New Schema types and Schema errors columns.

0.3.0 — 2026-10-06

  • Pro: crawls are saved automatically and listed under Saved crawls (⌘O / Ctrl+O): reopen, rename, delete, and continue a crawl that was stopped or hit a URL limit from exactly where it left off. Save as a name with ⌘S / Ctrl+S.
  • Continue button after a stopped or capped crawl (Pro removes the free limit for the continuation).
  • Pro: JavaScript rendering. Pages are rendered in headless Chromium (your installed Chrome, or a one-click download) and compared with the HTML response: new JavaScript tab and eight checks for titles, descriptions, canonicals, robots directives, H1s and links that only exist after rendering, content that depends on JavaScript, and pages that failed to render. The detail pane shows what changed and the rendered DOM.
  • “What to fix first”: the site health score explained, and issues ranked by the points each fix recovers, weighted towards your best-linked pages, with effort hints and a jump to the affected URLs (⇧⌘P / Ctrl+Shift+P). Free shows the top three, Pro the full list; the PDF report uses the same ranking.
  • Pro: PDF audit report (Export → PDF audit report): cover, executive summary with health score and key numbers, a ranked “fix these first” list, every issue with why it matters, how to fix it and example URLs, and a crawl appendix. White-label with your name, logo, accent colour and prepared-for line under Settings → Report.
  • Pro: compare two saved crawls of a site: health and issue counts side by side, URLs added, removed or changed, and per-URL field changes (status, title, canonical, indexability and more) with issues fixed or introduced.

0.2.1 — 2026-10-04

  • Windows 10 and 11 build (64-bit installer), available from crawlune.com/download. Not yet code-signed; SmartScreen may warn until the certificate is issued.
  • Linux build: AppImage for any distribution and a .deb for Debian and Ubuntu (64-bit).
  • Keyboard shortcut hints show Ctrl on Windows and Linux.
  • Free plan now crawls up to 550 URLs per crawl (was 500).

0.2.0 — 2026-10-03

  • Pro licences: paste the key from your crawlune.com account in Settings → Licence to unlock unlimited crawls. Two devices per seat, works offline for 30 days between checks; deactivate from the app or the account page.

0.1.2 — 2026-09-29

  • Update check: the app looks for a newer version a few seconds after launch and shows an “Update to x.y.z” button in the toolbar; About has “Check now”.
  • About dialog with version, what’s new, feedback and legal links.
  • “Learn more” on every issue explanation opens the check’s page on crawlune.com.
  • Window size and position are remembered between launches.
  • Starting a new crawl asks for confirmation when it would discard a finished one.
  • Send feedback (feedback@crawlune.com) from About and the command palette.

0.1.1 — 2026-09-28

  • macOS only for now: signed and notarised builds for Apple Silicon and Intel, distributed from crawlune.com/download.
  • Prices shown in US dollars.

0.1.0 — 2026-09-26 (first public beta)

First release of the free desktop crawler for macOS, Windows and Linux.

Crawling

  • Breadth-first crawl of HTML, images, CSS and JavaScript with pause, resume and stop.
  • Spider mode from a start URL, or list mode from pasted URLs.
  • Configurable concurrency, delay, timeout, depth, subdomain and external scope, include/exclude regex, robots.txt handling and user-agent presets.
  • Sitemap discovery from robots.txt and common sitemap paths; llms.txt and AI-crawler rules read from robots.txt.
  • Free plan crawls up to 500 URLs per crawl.

Audit

  • 106 checks across response codes, URLs, security headers, titles, descriptions, headings, canonicals, directives, content, images, links, sitemaps, hreflang and pagination.
  • Every check explains why it matters and how to fix it; the same text is published at crawlune.com/checks.
  • Site health score and overview counts.

Workspace

  • 16 category tabs with filters, sortable virtualised table, column chooser, search and issue filtering from the sidebar.
  • Detail pane with details, issues, inlinks, outlinks, images, resources, response headers and raw HTML with find.
  • Right-click menus, keyboard navigation and a command palette.
  • CSV export of any view, XML sitemap generator, dark and light themes.

Known limitations

  • No JavaScript rendering: pages are audited from server HTML.
  • Crawls are not saved between sessions.
  • Windows and Linux builds are not published yet.