My Reddit AMA on the Law of Web Scraping

Eric Goldman's Reddit AMA on the Law of Web Scraping, held on r/WebScrapingInsider, highlighted recent court rulings that tighten liability for unauthorized data extraction and clarified how the Computer Fraud and Abuse Act (CFAA) is being applied to modern scraping tools. For website owners, this means that any AI bot that harvests content without explicit permission now faces a higher risk of being deemed illegal under the CFAA, especially if the site’s robots.txt or llms.txt files explicitly deny access. Courts are increasingly looking at the intent and the technical means of the scraper, so bots that ignore exclusion directives can trigger legal action, fines, or forced takedowns. Owners should treat llms.txt as a legally enforceable policy rather than a courtesy, and ensure it accurately reflects the data they wish to protect. **Actionable tips:** 1. Audit your llms.txt file today and add precise disallow rules for AI endpoints (e.g., `User-agent: * Disallow: /api/v1/`). 2. Implement real‑time bot detection that flags requests violating llms.txt and automatically serves a 403 response, logging IPs for potential cease‑and‑desist letters. 3. Publish a clear “No Scraping” policy on your site’s legal page and reference the llms.txt file, giving you a stronger legal footing if you need to pursue litigation against offending bots.