Obituary: Farewell to robots.txt (1994-2025)

Meta's announcement of the retirement of robots.txt, a standard used for over three decades to control how web crawlers interact with websites, marks a significant shift in the way website owners manage their online presence. The retirement of robots.txt, which was born out of necessity in 1994 when Martijn Koster's server crashed under a faulty crawler, signals the end of an era in digital civility. This means that website owners will need to adapt to new methods for controlling web crawler access, such as using alternative protocols or updating their website's architecture to accommodate the change. The retirement of robots.txt may lead to increased AI bot traffic, as crawlers may no longer be restricted by the file's directives, potentially impacting website performance and security. To prepare for this change, website owners should take the following steps: review their website's current robots.txt file and update their access control methods accordingly, consider implementing alternative protocols such as meta tags or HTTP headers to control crawler access, and monitor their website's traffic and performance closely to detect any potential issues related to the retirement of robots.txt. Additionally, they should also update their llms.txt files to ensure that AI bots are properly managed and directed on their websites.