Should I Block AI Crawlers Or Measure Their Value First? – Ask An SEO via @sejournal, @HelenPollitt1

Search Engine Journal's discussion on whether to block AI crawlers or measure their value first highlights the delicate balance between conserving server resources and maintaining online visibility. The article notes that AI bots can drain server resources and scrape IP addresses, but blocking them outright may result in invisibility in Large Language Model (LLM) answers, potentially harming a website's search engine ranking. This means that website owners must carefully consider the implications of blocking AI crawlers, as it may impact their website's discoverability and credibility. With the increasing use of LLMs in search engines, website owners who block AI crawlers may inadvertently reduce their online presence, making it essential to weigh the costs and benefits of blocking these bots. To navigate this challenge, website owners can take the following steps: monitor their server resources to identify the impact of AI crawlers, analyze their website's traffic to determine the value of AI bot traffic, and update their llms.txt files to selectively allow or block specific AI crawlers, ensuring they maintain a balance between resource conservation and online visibility.