Understand this story
On October 30, Google announced plans to support a shared AI training control signal in robots.txt, allowing website owners to express their training preferences to multiple AI vendors in a unified way, and said the goal is to implement it by 2027. The mechanism is based on the work of the IETF AI Preferences working group, writing the __usage_ai-training policy token in the User-Agent line without adding new directive names or syntax; an opt-out declaration that does not specify Google-Extended will be treated as prohibiting its use. Subsequent reports said Google plans to support the unified policy token starting in 2027, so that website operators can use a single entry to express to multiple AI vendors participating in the mechanism whether content is allowed to be used for model training, without having to maintain vendor-specific rules such as GPTBot, ClaudeBot, and Google-Extended one by one.
AI-generated from public reports
Latest development · Report
Google plans to support a unified AI training policy token in robots.txt starting in 2027Developments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 2
heise online
Google plans to support a unified AI training policy token in robots.txt starting in 2027
Google plans to support a unified policy token in robots.txt (__usage_ai-training) starting in 2027, allowing website operators to use a single entry to express, to multiple AI vendors participating in the mechanism, whether content may be used for model training, without having to maintain vendor-specific rules such as GPTBot, ClaudeBot, and Google-Extended one by one. The proposal is based on prior work by the IETF AIPREF working group and follows existing User-Agent syntax; parsers that do not recognize the token will ignore it as an unknown User-Agent under RFC 9309. Google says that if a site prohibits AI training via this token without separately specifying Google-Extended, it will automatically be treated as having blocked Google-Extended; and explicitly allowing Google-Extended is also a prerequisite for a site's content to appear in Grounding answers in the Gemini app.
heise onlineRelated discussion
heise online joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
cnBeta
OpenAI, Anthropic, Google, and Microsoft jointly draft a statement rejecting AI training
OpenAI, Anthropic, Google, and Microsoft have jointly drafted an internet draft: websites only need to add User-Agent: __usage_ai-training and Disallow:/ to robots.txt to declare that all content on the site is prohibited from being used for AI training, and they can also set permissions or prohibitions by directory. The companies' support timelines differ: Microsoft says it will comply starting February 1, 2027, Google plans to support it in early 2027, OpenAI says it will implement it within the next few months, and Anthropic appears to have already begun supporting it. This statement only expresses a content usage preference and will not directly block crawler access; restricting crawling still requires traditional robots.txt rules.
gihyo.jp
OpenAI, Google, and Microsoft will support the __usage_ai-trainin… identifier in robots.txt
On October 30, 2026, OpenAI, Google, and Microsoft announced that they will support using the __usage_ai-training identifier in robots.txt to uniformly express to multiple AI vendors whether content may be used for training. This mechanism is based on the draft specification Machine-Readable Training Preference Signal, published after discussions in the IETF AIPREF, and was submitted by Anthropic's Kevin Kelley; it is not an IETF-approved standard. The three have different implementation timelines: Google plans to support it in early 2027, Microsoft will apply it starting February 1, 2027, and OpenAI says it will implement it within the coming months; Anthropic has also begun implementation.
Oct 31
Unite.AI
Google plans to support shared AI training control signals in robots.txt
On October 30, Google announced plans to support shared AI training control signals in robots.txt, allowing website owners to express training preferences to multiple AI vendors in a unified way, and said the goal is to implement it by 2027. The mechanism is based on the work of the IETF AI Preferences working group, writing a __usage_ai-training policy token in the User-Agent line, without adding new directive names or syntax; opt-out declarations that do not specify Google-Extended will be treated as prohibiting its use. OpenAI committed the same day to honor the signal and plans to implement it within the coming months, while Microsoft announced that starting February 1, 2027, it will replace NoArchive and NoCache with the new preference.
Heat trend for this story
- Current
- 17
- Period peak
- 17 Nov 2 23:00
- 24-hour change
- —
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.