Understand this story
English text is being prepared.
AI-generated from public reports
Latest development · Report
OpenAI, Anthropic, Google, and Microsoft jointly draft a statement rejecting AI trainingDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 2
cnBeta
OpenAI, Anthropic, Google, and Microsoft jointly draft a statement rejecting AI training
OpenAI, Anthropic, Google, and Microsoft have jointly drafted an internet draft: websites only need to add User-Agent: __usage_ai-training and Disallow:/ to robots.txt to declare that all content on the site is prohibited from being used for AI training, and they can also set permissions or prohibitions by directory. The companies' support timelines differ: Microsoft says it will comply starting February 1, 2027, Google plans to support it in early 2027, OpenAI says it will implement it within the next few months, and Anthropic appears to have already begun supporting it. This statement only expresses a content usage preference and will not directly block crawler access; restricting crawling still requires traditional robots.txt rules.
English text is being prepared.
OpenAI, Google, Microsoft and Anthropic jointly promote AI data scraping opt-out standard
OpenAI, Google, Microsoft and Anthropic announced that they will adopt a new standard that allows content publishers to declare their content usage preferences to multiple AI companies at once via robots.txt, without having to set them individually for each company. The standard was proposed to the IETF by Anthropic technical lead Kevin Kelley. Anthropic has already adopted it first, OpenAI plans to implement it within the next few months, and Google and Microsoft are expected to support it next year. Meta spokesperson Andy Stone said he supports establishing such a universal signal, but did not specify an implementation plan.
IT Home · AIRelated discussion
IT Home · AI joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Oct 31
Unite.AI
Google plans to support shared AI training control signals in robots.txt
On October 30, Google announced plans to support shared AI training control signals in robots.txt, allowing website owners to express training preferences to multiple AI vendors in a unified way, and said the goal is to implement it by 2027. The mechanism is based on the work of the IETF AI Preferences working group, writing a __usage_ai-training policy token in the User-Agent line, without adding new directive names or syntax; opt-out declarations that do not specify Google-Extended will be treated as prohibiting its use. OpenAI committed the same day to honor the signal and plans to implement it within the coming months, while Microsoft announced that starting February 1, 2027, it will replace NoArchive and NoCache with the new preference.
Testing Catalog
October 31 AI Daily Brief: Anthropic subscription renamed, Grok Bot 1.0 released
Anthropic renamed Max 5x to Max ($100) and Max 20x to Ultra ($200), and the verified Ultra version removed the 5-hour limit; the maximum request size for the Claude API increased from 32 MB to 128 MB. The first stable version of xAI's Grok Bot 1.0 is available for download, with new per-bot voice and voice language settings. NVIDIA open-sourced the 9.6M-parameter voice cleanup model RE-USE, and Pika released the open-weight video model H3 Flex.
Jason ZhuRelated discussion
Jason Zhu joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
SiliconANGLE
Google, Microsoft, Anthropic and OpenAI to support new AI crawler opt-out standard
Google, Microsoft, Anthropic and OpenAI announced they will support an in-development website crawler standard that lets website operators block all crawlers collecting training data for generative AI models with a single robots.txt entry; the setting does not apply to models performing only non-generative tasks. The four companies have submitted a draft to the IETF, which maintains the Robots Exclusion Protocol, and Microsoft will also provide a setting called AI Grounding to block web access requests made by models to answer user questions. Microsoft and OpenAI plan to support the standard within the next few months, Google will follow next year, and Meta said it supports the idea but has not yet committed to implementing it.
Thurrott
Anthropic, Google, Microsoft and OpenAI agree to let websites opt out of AI training
Anthropic, Google, Microsoft and OpenAI have agreed to support and implement an opt-out mechanism developed by an IETF working group, allowing website owners to express their preferences regarding the use of their content for generative AI training, separately from search indexing and archiving. OpenAI calls it a voluntary mechanism that lets publishers convey the same preference to multiple participating vendors at once; Microsoft says publishers should not have to choose between Bing discoverability and models using their work. Anthropic has already implemented the mechanism, while Google, Microsoft and OpenAI will implement it in 2027, after which it will be proposed as an international standard.
The Verge AI
OpenAI, Google, Microsoft, and Anthropic will adopt a unified AI crawling opt-out standard
OpenAI, Google, Microsoft, and Anthropic said they will adopt a new standard that lets publishers express their opt-out preferences to multiple AI companies at once. The shared policy token signal was proposed at the IETF by Kevin Kelley, Anthropic's standards technical lead, and will be written into websites' robots.txt files; currently, websites must set preferences separately for each AI company. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it within the next few months; Microsoft will also offer more granular controls, allowing publishers to opt out of AI training or AI grounding separately, while their sites will still appear in Bing search results.
Unite.AI
OpenAI announces it will honor the shared robots.txt signal to express AI training preferences
On October 30, 2026, OpenAI announced that it will honor the shared signal in robots.txt, allowing website operators to express the same AI training preference to multiple participating vendors at once, with implementation planned over the coming months. The signal is defined in an Internet-Draft submitted by Anthropic's Kevin Kelley; it uses the __usage_ai-training policy token bound to paths, adds no new directive names or syntax, and follows the parsing rules of RFC 9309. OpenAI stated that existing controls such as GPTBot and OAI-SearchBot remain in effect, and encouraged other AI vendors to honor the signal as well.
Heat trend for this story
- Current
- 13
- Period peak
- 50 Oct 31 11:00
- 24-hour change
- -49%
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.