LLMS Central - The Robots.txt for AI
Web Crawling

97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows via @sejournal, @MattGSouthern

Search Engine Journal2 min read
Share:
97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows via @sejournal, @MattGSouthern

Original Article Summary

Ahrefs analyzed 137K domains and found 97% of llms.txt files received zero requests. AI retrieval bots accounted for 1% of total requests. The post 97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows appeared first on Search Engine Journal.

Read full article at Search Engine Journal

Our Analysis

Ahrefs' analysis of 137K domains revealing that 97% of llms.txt files received zero requests sheds light on the current state of AI bot traffic and llms.txt file usage. The finding that AI retrieval bots accounted for only 1% of total requests suggests that the adoption and utilization of llms.txt files are still in their infancy. This means that website owners have a unique opportunity to get ahead of the curve and optimize their llms.txt files to effectively manage AI bot traffic. With the majority of llms.txt files not being requested, website owners can take proactive steps to ensure their files are properly configured and up-to-date, potentially gaining a competitive edge in terms of AI bot management. To take advantage of this situation, website owners can follow these actionable tips: (1) review and update their llms.txt files to ensure they are accurately reflecting their website's content and structure, (2) monitor their website's traffic and analytics to identify any potential AI bot traffic patterns, and (3) consider using tools like Ahrefs to analyze and optimize their llms.txt files for better performance.

Related Topics

BotsSearch

Track AI Bots on Your Website

See which AI crawlers like ChatGPT, Claude, and Gemini are visiting your site. Get real-time analytics and actionable insights.

Start Tracking Free →