Google, Microsoft, Anthropic and OpenAI to support new AI crawler opt-out standard
Google, Microsoft, Anthropic and OpenAI announced they will support an in-development website crawler standard that lets website operators block all crawlers collecting training data for generative AI models with a single robots.txt entry; the setting does not apply to models performing only non-generative tasks. The four companies have submitted a draft to the IETF, which maintains the Robots Exclusion Protocol, and Microsoft will also provide a setting called AI Grounding to block web access requests made by models to answer user questions. Microsoft and OpenAI plan to support the standard within the next few months, Google will follow next year, and Meta said it supports the idea but has not yet committed to implementing it.
#English text is being prepared.#English text is being prepared.#Google