In This Article
Microsoft has joined Google, OpenAI and Anthropic in backing a shared way for publishers to signal that their websites should not be used to train generative AI models. The proposed robots.txt instruction could simplify settings across providers, but it is voluntary, not yet an adopted internet standard, and does not itself block crawling.
Key takeaways
The announcement gives website owners a common preference to communicate, while leaving important differences in timing and scope between providers. For enterprise teams, the distinction between training, crawling and AI-generated answers will shape how the change affects content strategy.
- The shared robots.txt token is
__usage_ai-training. - Microsoft plans to honor it from February 1, 2027; Google targets early 2027, while OpenAI says implementation is coming in the coming months.
- Anthropic says it has begun implementing support, but has not set a completion date.
- The signal does not automatically prevent crawling, guarantee payment or undo prior data collection.
One instruction for participating providers
Publishers currently need separate crawler rules for different AI companies. Under the proposal, they could use a single policy token in a robots.txt group to express whether content may be used for AI training, including setting different preferences for selected website paths. Microsoft, Google and OpenAI announced plans to support the signal on October 30, 2026; Anthropic shared its commitment through the standards working group.
The mechanism is based on an individual Internet-Draft submitted to the Internet Engineering Task Force’s AI Preferences working group. It has no formal standing as an IETF standard and may change. The draft also specifies that a general wildcard rule such as User-agent: * does not count as expressing a preference through this new signal.
Training is not crawling or grounding
The token expresses a preference about using content to change a generative model’s learned parameters. It does not instruct every crawler to stay away, nor does it replace provider-specific access rules. Publishers seeking to prevent crawling still need to manage those rules separately, and the proposal cannot reverse collection that has already occurred.
Microsoft is also introducing separate page-level controls for AI grounding—the use of web content to inform generated answers. Its training and grounding preferences are distinct, while Google says it will interpret the shared training opt-out as a Google-Extended disallow. Google-Extended also covers grounding for Gemini services, so Google’s handling may affect answer-time use as well as training. Microsoft says restrictions do not affect traditional Bing search eligibility; neither company guarantees visibility in AI answers.
What publishers and enterprise teams should watch
Microsoft’s transition includes a December 15, 2026, opening date for temporary grounding preferences in Bing Webmaster Tools for eligible sites using its older NoArchive or NoCache controls. Microsoft plans to stop treating those tags as AI preferences after March 1, 2027. Website owners will need to review their settings and timing, particularly because preferences are applied as Microsoft recrawls pages.
For enterprises, the announcement underscores the value of clear content governance: teams need to know which material is published publicly, how it may be used, and which controls apply across platforms. That is separate from governing internal analytics data. Spargent helps organizations build and manage Microsoft Fabric and Power BI data platforms, supporting reliable reporting and decision-making as digital policies evolve. The shared signal may reduce repetitive website configuration, but it does not settle copyright disputes, create licensing terms or establish compensation.
- Anthropic, Google, Microsoft, and OpenAI Agree to Let Sites Opt Out of AI Training, Thurrott.com.
- Google, Microsoft, OpenAI and Anthropic back one AI training opt-out, PPC Land.
- AI giants promise publishers new way to block some scraping, Tech Xplore.
- OpenAI, Google, Microsoft And Anthropic Back Shared AI Training Opt-Out For Publishers, The 420.in.