Publishers say no to AI scrapers, block bots at server level

Publishers' decision to block AI scrapers at the server level to prevent their work from being used as training data and to avoid server overload marks a significant shift in the way websites manage automated traffic. This move is likely to impact website owners who rely on open web data for their AI models, as they will need to find alternative sources of training data or risk being blocked by an increasing number of publishers. Website owners who use AI scrapers will need to reassess their data collection strategies to avoid being cut off from valuable information. To adapt to this change, website owners can take several steps: firstly, they should review their llms.txt files to ensure they are not inadvertently blocking legitimate traffic; secondly, they should consider implementing more selective bot-blocking measures to allow desired AI traffic while keeping out unwanted scrapers; and thirdly, they should explore alternative data sources, such as partnerships with publishers or paid data providers, to reduce their reliance on scraped data.