Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern

Google’s Illyes clarified that robots.txt “covers crawling, not how content gets used for AI training,” and urged site operators to percent‑encode non‑English paths in their robots.txt files. This distinction means that while you can still block crawlers from accessing pages, you cannot rely on robots.txt to stop large language models from scraping and ingesting your content for training purposes. For website owners, the implication is a heightened risk of unintended data exposure to AI services that ignore robots directives, potentially affecting copyright, brand safety, and SEO performance. To protect your site, first audit your robots.txt for any non‑ASCII URLs and apply percent‑encoding (e.g., /%E6%96%87%E5%AD%97) to ensure proper interpretation by all crawlers. Second, supplement robots.txt with a robust llms.txt file that explicitly states whether your content may be used for AI training, leveraging the emerging llms.txt standard to convey licensing terms to model developers. Finally, monitor inbound AI bot traffic using llmscentral’s bot‑tracking dashboard, setting alerts for IP ranges associated with major AI providers to detect unauthorized scraping early.