Microsoft Will Honor a Shared AI Opt-Out Token on 1 February 2027, but the Standard Behind It Is Still One Engineer’s Draft
Microsoft will start honoring a shared robots.txt token for AI training on 1 February 2027. Google and OpenAI are later, and the IETF draft behind it is still an individual submission.
Microsoft will begin honoring a shared robots.txt token for AI training preferences on 1 February 2027, the company said on 30 October. The token, __usage_ai-training, lets a publisher signal a training opt-out to several AI providers with one rule instead of a separate rule for every crawler.
OpenAI, Google and Anthropic joined the announcement. Their calendars do not match. Anthropic has started implementing the signal, OpenAI says it will arrive in the coming months, and Google says 2027. Microsoft is the only company with a firm enforcement date.
The catch sits in the paperwork. The token comes from an Internet-Draft written by Kevin Kelley, a standards lead at Anthropic, and submitted as an individual draft on 30 October. It is not an adopted working group standard, it expires on 3 May 2027, and the IETF AI Preferences group is still arguing over the rest of the vocabulary.
Quick answer: One robots.txt group can now tell Microsoft, Google, OpenAI and Anthropic not to train on your pages. But the promise is voluntary, the dates run from February 2027 into late 2027, the token covers training only, and none of it removes content that has already been crawled.
What the token actually does
The draft adds no new directive names and no new grammar. It reuses the group structure, path matching and longest-match rule already defined for robots.txt in RFC 9309. A publisher writes a normal group, with the policy token in the user agent line.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example says no training on the site except under /press-kit/. Because the token is a product token, a crawler that does not implement it ignores the group, exactly as it would ignore a group for an unknown bot.
Two details matter. A wildcard group, User-Agent: *, must not be read as a training preference, so a site that blocks everything for all bots has not opted out of training by accident. And a file with no group for the token expresses no preference at all.
The draft is also careful about what it is not. Section 4.4 says the policy token is not a directive for crawling behavior. A provider may use the preference to decide whether to fetch a page, but the document itself does not order a block. The enforcement comes from each company’s own promise.
Four companies, four calendars
Microsoft published the most detailed plan. Its new training and grounding preferences replace the older NoArchive and NoCache tags. Temporary grounding controls open in Bing Webmaster Tools on 15 December 2026, enforcement begins on 1 February 2027 as pages are recrawled, and publishers using the old tags have until 1 March 2027 to move. Temporary grounding preferences expire on 30 September 2027.
| Company | What it will honor | Stated timing | Existing control still needed |
|---|---|---|---|
| Microsoft | Shared training token plus separate page-level training and grounding preferences | 1 February 2027 | NoArchive and NoCache until March 2027 |
| Shared token, read as a block on Google-Extended unless that token is named | 2027 | Google-Extended |
|
| OpenAI | Shared token alongside its current crawler rules | Coming months, no date | GPTBot and OAI-SearchBot |
| Anthropic | Shared token, implementation reported to be under way | Started, no completion date | ClaudeBot |
Google’s post adds a useful piece of interpretation. If a site uses the shared token to opt out and does not mention Google-Extended, Google plans to treat that as a disallow for Google-Extended. A site that explicitly allows Google-Extended stays eligible for grounding in Gemini Apps, and that token does not affect inclusion in Search. The older token remains a more specific instruction, not a redundant one.
OpenAI was the vaguest. It aims to implement the signal in the coming months and will publish details later, and says the new signal will not replace its existing crawler controls.
Anthropic’s status is the least documented. The Verge reported that Anthropic has already adopted the signal, and the draft’s author works there, but Anthropic has not published a dated rollout note on its crawler help page. Treat “already started” as reported, not confirmed by an Anthropic document.
Microsoft is joining industry peers in announcing a shared AI training preference in robots.txt – and going further with granular page-level training preferences and a separate grounding preference for publishers and site owners. See: https://t.co/ZV1oq5ML27
— Microsoft Bing Webmaster Team (@BingWMC) October 30, 2026
The catch is in the standards process
Microsoft calls the draft “significant progress that merits adoption” and admits “additional work remains.” Google calls it a draft proposal and says discussions continue. Those are honest descriptions of a document that one person submitted, not a finished standard.
That matters for anyone building a compliance story on top of it. An individual Internet-Draft can change, be replaced or expire. The working group has separate documents for vocabulary and for attaching preferences to HTTP content, and both are still in progress.
There is a second limit. The token covers AI training as the draft defines it: using an asset to modify the learned parameters of a generative model. It does not cover grounding, the live retrieval that feeds chatbot answers. Microsoft created a separate grounding preference, and Google folds grounding into Google-Extended. A publisher who opts out of training may still be quoted in an AI answer.
What publishers should change, and what not to
Do not delete the company specific groups yet. The four implementations start at different times, and a site that removes GPTBot or ClaudeBot because it added the shared token could open a gap during 2027. The draft says company specific tokens can be more specific and should take precedence.
For Microsoft, the urgent work is the tag migration. A site that relies on NoArchive or NoCache to limit training or grounding loses that signal after 1 March 2027. The replacement is a training preference in robots.txt or at page level, plus a separate grounding preference. Microsoft says the page level training setting overrides the robots.txt setting. For everyone else, add the shared group now and keep the old groups in place until the crawlers catch up.
Confirmed vs Not Confirmed
| Claim | Status | Basis |
|---|---|---|
The token is __usage_ai-training and uses ordinary robots.txt groups |
Confirmed | IETF draft, 30 October 2026 |
| Microsoft enforces its new preferences from 1 February 2027 | Confirmed | Microsoft Bing blog and Webmaster Tools FAQ |
| Google treats the token as a Google-Extended disallow when that token is absent | Confirmed | Google Search Central blog, 30 October 2026 |
| OpenAI will implement “in the coming months” | Confirmed as a statement, date not given | OpenAI announcement, 30 October 2026 |
| Anthropic has already started implementing support | Not confirmed | Reported by The Verge, no Anthropic page with a date |
| Meta will honor the token | Not confirmed | Meta’s Andy Stone supported the idea but did not commit |
| The token removes content already used in training | False | Draft applies at retrieval time; no retroactive promise |
What it means for Indian developers
If you run a publishing site, the token is a cheap addition and a weak guarantee. It reduces the maintenance of listing every crawler, but it does not create a licence, a payment or a right to audit. Those still come from contracts, and from separate efforts such as the Really Simple Licensing standard and Cloudflare’s crawler blocks.
Frequently asked questions
Does the token stop AI training on my site today?
No. Anthropic is the only company reported to have started, and it has no published completion date. Microsoft starts on 1 February 2027, Google says 2027, and OpenAI says the coming months.
Does it remove content that is already in a training set?
No. The draft ties a preference to the robots.txt file in effect when a page was retrieved. None of the four companies has said the token applies to data already collected.
Should I keep Google-Extended, GPTBot and ClaudeBot rules?
Yes, for now. The implementations are staggered through 2027, and Google says a direct rule for Google-Extended is treated as more specific. Removing the old groups early creates a gap that the shared token may not fill.
Is this an official IETF standard?
No. It is an individual Internet-Draft published on 30 October 2026 and set to expire on 3 May 2027. The working group is still developing its vocabulary and attachment documents, and the draft may change before anything is adopted.
Sources
- Kevin Kelley, “Machine-Readable Training Preference Signal,” IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, 30 October 2026. datatracker.ietf.org
- Krishna Madhavan, “Your content, your choice,” Microsoft Bing Blog, 30 October 2026. blogs.bing.com
- Microsoft, “New AI Preferences,” Bing Webmaster Tools help. bing.com
- John Mueller, “Supporting a shared AI training control for websites,” Google Search Central Blog, 30 October 2026. developers.google.com
- OpenAI, “Making it easier for websites to express AI training preferences,” 30 October 2026. openai.com
- Emma Roth, “The biggest names in AI are backing a new way to opt out of scraping,” The Verge, 30 October 2026. theverge.com
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” Claude Help Center, April 2026. support.claude.com
Related on AI Magazine: Cloudflare’s 13 day cache leak, the UK probe into Meta’s AI content terms, and OpenAI’s GPT-6.1 pricing and system card.