Google, Microsoft, OpenAI and Anthropic back one AI training opt-out for websites
A draft rule would let sites block generative AI training with one robots.txt entry, The Next Web reported. Rollout dates vary, and the IETF has not endorsed the draft.
Google, Microsoft, OpenAI and Anthropic will honour a single shared signal that lets websites keep their content out of generative AI training, according to The Next Web.
The signal comes from an IETF draft by Kevin Kelley of Anthropic, published on Friday, the outlet reported. Google, OpenAI and Microsoft announced their support the same day. The signal works through robots.txt, the file sites already use to tell crawlers what they may fetch. A site adds a group for a new "policy token", __usage_ai-training, and then lists the paths it does not want used for training.
This matters because a site that wants to opt out today has to write separate rules for each company's crawler, The Next Web reported.
“Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”Krishna Madhavan, principal product manager for Microsoft Bing
The companies are on different timelines. Kelley wrote that Anthropic has already started work on the signal and that it would complement Anthropic's existing crawler controls rather than replace them. OpenAI said it aims to implement it "in the coming months" and that controls such as GPTBot will still apply. Google said it aims to support the signal in 2027, and its crawler documentation says early 2027. Microsoft will honour the new preferences from 1 February 2027. They will replace the NoArchive and NoCache tags it has used since 2023.
Microsoft is also adding a separate page-level setting for grounding, which is when an AI answer draws on a live web page. The company said the new preferences "do not establish licensing or payment arrangements". It also said blocking grounding "will significantly reduce your content's visibility in AI-generated experiences".
According to The Next Web, the draft describes the token as a statement of preference about how content is used, not an order about crawling. Private agreements between a site and an AI company can override it. The draft is an individual submission. The IETF's datatracker says the IETF has not endorsed it and that it has no formal standing in the standards process.
Google said the proposal is consistent with the EU's Code of Practice for general-purpose AI under the AI Act.
Several questions remain open. Kelley expected problems along the way:
“It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover.”Kevin Kelley, Anthropic
It is also unclear whether other AI companies will sign on. The Next Web, citing The Verge, reported that Meta communications head Andy Stone voiced support for a common signal on X but did not say whether Meta would honour it.
The signal does not address payment, which publishers have pushed for separately. The Next Web noted that in July, Cloudflare told AI crawlers to pay or get blocked.
