Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop

RayenMlayeh's release of Jevcrawl, a web crawler for AI agents that operates without an LLM in the crawl loop, promises to find the answer page twice as often as Crawl4AI and at a cost ten times lower than using Claude to crawl. For website owners, this means AI‑driven traffic will become more efficient and less resource‑intensive, increasing the volume of bots that can scrape content without the overhead of large language models. Sites that rely on ad revenue or have strict content licensing may see a surge in legitimate‑looking queries that bypass traditional bot‑detection heuristics, while the cheaper cost per answer encourages more frequent crawling of deeper site layers. **Actionable tips:** 1. Update your llms.txt file to explicitly list Jevcrawl’s user‑agent strings (e.g., `User-Agent: Jevcrawl/*`) and define allowed/disallowed paths, ensuring you control which sections AI agents can access. 2. Deploy a lightweight request‑rate monitor that flags spikes from Jevcrawl’s IP ranges, allowing you to throttle or challenge excessive crawls without affecting human visitors. 3. Leverage llmscentral’s bot‑tracking dashboard to differentiate Jevcrawl traffic from other AI crawlers, giving you granular insight for rate‑limiting or monetization decisions.