Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

OpenAI's rollout of a new website‑crawling standard, backed by Google, Microsoft, Anthropic and others, lets publishers add a single `llms.txt` line to opt‑out of AI training while still permitting regular indexing. For website owners this means their content can remain discoverable by search engines but be excluded from large‑scale model training datasets. The distinction is crucial: while Google’s and Microsoft’s crawlers will still fetch pages for SEO, they must now respect the `llms.txt` directive and refrain from feeding the data into OpenAI‑ or Anthropic‑powered models. Failure to implement the line could result in inadvertent inclusion in future LLMs, raising copyright and brand‑representation concerns, especially for sites with proprietary, copyrighted, or sensitive material. **Actionable tips:** 1. Add the exact `llms.txt` entry — `llms: training=off` — to your site’s root directory and verify it with the OpenAI compliance checker. 2. Use llmscentral.com’s monitoring dashboard to track which major bots (Googlebot, Bingbot, ClaudeBot) are honoring the directive, flagging any violations for immediate follow‑up. 3. Regularly audit your server logs for unexpected AI‑specific user‑agents and update your `robots.txt` to block them if they ignore the `llms.txt` opt‑out.