LLMS Central - The Robots.txt for AI
AI Models

Show HN: 1,011 AI crawler requests. Google Analytics saw zero

Adwait.me1 min read
Share:
Show HN: 1,011 AI crawler requests. Google Analytics saw zero

Original Article Summary

Google Analytics can't see GPTBot or ClaudeBot. Here's how I built a server-side tracker that can — and what I found in 72 hours. Comments URL: https://news.ycombinator.com/item?id=47386037 Points: 1 # Comments: 0

Read full article at Adwait.me

Our Analysis

Adwait's development of a server-side tracker to detect AI crawler requests, such as those from GPTBot or ClaudeBot, which are not visible in Google Analytics, highlights a significant blind spot in website analytics. This means that website owners may be underestimating the amount of AI-generated traffic to their sites, as traditional analytics tools like Google Analytics are not equipped to track these requests. The fact that Adwait's server-side tracker detected 1,011 AI crawler requests in just 72 hours, while Google Analytics saw zero, underscores the magnitude of this issue. To address this issue, website owners can take several steps: firstly, consider implementing a server-side tracker like Adwait's to get a more accurate picture of AI-generated traffic; secondly, review and update their llms.txt files to reflect the growing number of AI crawlers like GPTBot and ClaudeBot; and thirdly, monitor their website's traffic patterns and adjust their content strategies accordingly to account for the increasing presence of AI bots.

Related Topics

ClaudeGoogleWeb CrawlingBots

Track AI Bots on Your Website

See which AI crawlers like ChatGPT, Claude, and Gemini are visiting your site. Get real-time analytics and actionable insights.

Start Tracking Free →