OpenAI, Anthropic, Google, and Microsoft jointly draft a statement rejecting AI training
OpenAI, Anthropic, Google, and Microsoft have jointly drafted an internet draft: websites only need to add User-Agent: __usage_ai-training and Disallow:/ to robots.txt to declare that all content on the site is prohibited from being used for AI training, and they can also set permissions or prohibitions by directory. The companies' support timelines differ: Microsoft says it will comply starting February 1, 2027, Google plans to support it in early 2027, OpenAI says it will implement it within the next few months, and Anthropic appears to have already begun supporting it. This statement only expresses a content usage preference and will not directly block crawler access; restricting crawling still requires traditional robots.txt rules.
Why read this
AI score 62
The draft provides the robots.txt declaration syntax and lists the differing support timelines of the four companies, helping website administrators plan content licensing in advance.
Topics
Body · AI translation
The statement reads as follows:
# Site-wide prohibition rules
User-Agent: __usage_ai-training
Disallow:/
# Example of allowing specific directories
User-Agent: __usage_ai-training
Allow:/archive/
# Example of not allowing specific directories
User-Agent: __usage_ai-training
disallow:/archive/
# Here, allowing specific directories does not conflict with the site-wide prohibition, i.e., /archive/ is allowed but all other directories are prohibited
Adding this content to a website's robots.txt file declares that the site-wide content is prohibited from being used for training AI models. Website administrators can also modify the allowed or disallowed directories, for example, allowing some directories to be used for AI training while not allowing other content to be used for AI training.
Support timelines differ among the companies:
This proposal is still an Internet Draft, but Microsoft has stated that it will comply with the declaration starting February 1, 2027, so websites that do not want their content used for AI training can modify their files and add the relevant declaration now.
Google plans to provide support starting in early 2027, but the specific timing is pending completion of preparatory work. OpenAI says it will implement the proposal within the coming months, and Anthropic appears to have already begun supporting the draft.
Prohibiting training and citing content to generate answers are different uses:
The above declaration only expresses a website's choice regarding the use of its content. According to the draft, setting the above declaration alone will not directly prevent crawlers from accessing web pages; if a website wants to restrict specific crawlers from scraping content, it still needs to configure this through traditional robots.txt rules.
Training content and AI answers citing web pages are also different uses. Microsoft additionally provides AI Grounding controls for its own services, allowing websites to decide whether content may be cited when AI generates answers; this control targets specific pages and applies only to Microsoft services.
Google stated that if a website uses the above declaration to refuse AI training but does not explicitly set Google-Extended, Google plans to handle it according to the website's refusal of the Google-Extended rules. Google-Extended involves Gemini model training and also involves the use of web page references in some Gemini answers.