LLMS Central - The Robots.txt for AI
Web Crawling

Nearly two-thirds of tested websites fail every bot test

Help Net Security••2 min read
Share:
Nearly two-thirds of tested websites fail every bot test

Original Article Summary

Malicious bot activity increased 124% between July 2025 and June 2026, compared with 13.2% growth in human traffic. Traffic from AI agents and large language model crawlers rose 82.3% during the same period, according to DataDome’s State of Bot & Agent Securi…

Read full article at Help Net Security

✨Our Analysis

DataDome's report that nearly two‑thirds of tested websites fail every bot test, alongside a 124% surge in malicious bot activity between July 2025 and June 2026, highlights a rapidly worsening threat landscape for site operators. For website owners this means that traditional defenses are no longer sufficient; the rise in AI‑driven crawlers—up 82.3% in the same period—means more automated content scraping, credential stuffing, and SEO poisoning attempts are targeting sites that lack robust bot discrimination. The failure rate indicates that most sites are unintentionally allowing malicious agents to access pages, potentially harvesting data, inflating analytics, and degrading performance. **Actionable steps:** 1. **Implement granular llms.txt rules** that explicitly deny known malicious LLM crawlers (e.g., “User‑Agent: BadLLMBot Disallow: /”) while allowing reputable AI agents like OpenAI’s ChatGPT for legitimate indexing. 2. **Deploy real‑time bot scoring** from services like DataDome, integrating their API to flag traffic that exceeds the 124% growth benchmark and automatically challenge or block it. 3. **Monitor llms.txt compliance** using llmscentral’s bot‑tracking dashboard to verify that your directives are being respected and adjust deny‑list entries as new threat signatures emerge.

Related Topics

Web CrawlingBots

Track AI Bots on Your Website

See which AI crawlers like ChatGPT, Claude, and Gemini are visiting your site. Get real-time analytics and actionable insights.

Start Tracking Free →