Google, Anthropic and others to support new AI crawling opt-out standard

Google, Microsoft, Anthropic and OpenAI announced plans to support an upcoming website crawling standard that would let site operators block AI developers from using their content for training with a single robots.txt entry. The companies have submitted a draft proposal to the Internet Engineering Task Force to incorporate the standard into the Robots Exclusion Protocol, with Microsoft and OpenAI planning support in the coming months and Google next year. Microsoft will also offer an "AI Grounding" setting to block requests made to answer user questions rather than collect training data, while Meta expressed support for the idea but hasn't committed to implementing it.