OpenAI, Google, Microsoft and Anthropic jointly promote AI data scraping opt-out standard
OpenAI, Google, Microsoft and Anthropic announced that they will adopt a new standard that allows content publishers to declare their content usage preferences to multiple AI companies at once via robots.txt, without having to set them individually for each company. The standard was proposed to the IETF by Anthropic technical lead Kevin Kelley. Anthropic has already adopted it first, OpenAI plans to implement it within the next few months, and Google and Microsoft are expected to support it next year. Meta spokesperson Andy Stone said he supports establishing such a universal signal, but did not specify an implementation plan.
Why read this
AI score 68
The new standard replaces per-company declarations with a unified statement in robots.txt; readers can learn about each company's implementation timeline and the differences in Meta's stance.
Topics
Body · AI translation
OpenAI, Google, Microsoft and Anthropic Join Forces to Implement AI Data Scraping Opt-Out Standard
2026-11-01 22:19 IT Home - Haomiao
Thanks to IT Home user Miemieyang for the tip submission!
IT Home, November 1 news: OpenAI, Google, Microsoft and Anthropic recently announced that they will adopt a new standard through which content publishers can choose to refuse AI scraping of their website content. With this control mechanism, content publishers can declare their content usage preferences regarding AI model training to multiple AI companies at once, without having to set rules separately for each company.
The standard was proposed to the Internet Engineering Task Force (IETF) by Anthropic technical lead Kevin Kelley and will be integrated into websites' robots.txt files.
For a long time, robots.txt files have been used to tell web crawlers which website content may be scraped and which may not, covering scenarios including search indexing, web archiving and AI model training. Currently, websites must separately declare their AI training-related content usage preferences to each AI company individually. Against this backdrop, various AI crawlers continue to scrape website content and feed web materials into AI model training, leaving content publishers overwhelmed.
IT Home notes that websites currently need to declare training preferences to each company separately, and the new standard aims to simplify this process. Anthropic has already taken the lead in adopting it, OpenAI plans to implement it within the next few months, and Google and Microsoft are expected to support it next year. Meta spokesperson Andy Stone also expressed support for establishing such a common signal, but did not specify a concrete implementation plan.