CrawluneAI analysis
Crawlune Pro adds three jobs that need a language model: finding pages that say the same thing in different words, writing titles and meta descriptions for pages that lack good ones, and proofreading. You bring the provider. Page text goes only to the provider you configure, and with Ollama it never leaves your machine.
1. Choose a provider
| Provider | Default models | Notes |
|---|---|---|
| OpenAI | gpt-4o-mini · text-embedding-3-small | Key from platform.openai.com. Cheapest for large crawls. |
| Anthropic | claude-sonnet-5-5 | Key from console.anthropic.com. No embeddings API, so semantic analysis needs another provider. |
| Google Gemini | gemini-2.5-flash · text-embedding-004 | Key from aistudio.google.com. |
| Ollama | llama3.2 · nomic-embed-text | Local, free, private. Install from ollama.com and pull the two models; Crawlune talks to localhost:11434. |
Settings → Integrations → AI provider takes the key, optional model overrides and the number of pages per run (best-linked pages first, 200 by default). Test the connection sends one tiny request. The key is stored only on your machine.
2. Run the jobs
After a crawl, press the AI button in the toolbar.
- Semantic duplicates. Each indexable page's title, H1 and visible text are embedded; pairs with a cosine similarity of 0.90 or more are flagged as Semantic duplicate, listed on both pages in the detail pane. Pages whose embedding sits far from the site's centre (more than two standard deviations below the mean similarity) are flagged Off-topic for this site. The Content tab gets Semantic dupes and Topic fit columns. Near-duplicates by wording are a separate, free check that runs without AI.
- Titles and descriptions. For pages with title or description issues, the selected page, or the best-linked pages, the model writes a 45–60 character title and a 120–155 character description from the page content, in the page's language. Review the list, copy per page or export CSV. Nothing is changed on your site.
- Spelling and grammar. The model proofreads the visible text of each page in whatever language it is written, ignoring brand names, code and navigation, and returns up to 20 findings per page. They appear in the detail pane and as the Spelling or grammar mistakes check.
Cost and limits
- Semantic analysis sends about 1,500 tokens per page in batches of 32; 200 pages cost a fraction of a cent with OpenAI embeddings.
- Proofreading and metadata send one request per page, four at a time, up to the pages-per-run limit (metadata: 100 per run).
- Provider rate limits surface as a message in the app; wait a minute and run again. Pages that fail are reported, not silently skipped.
- Results live in the crawl and are saved with it in the library; they are not sent to crawlune.com.