For years, telling AI companies not to train on your website meant writing a separate rule for each company’s crawler, then hoping you had not missed one. On October 30, that changed, at least on paper. Google, Microsoft, OpenAI, and Anthropic all said they will honor a single shared signal in robots.txt that says, in effect, do not train on this.
The signal is a policy token called __usage_ai-training. It comes from a new Internet-Draft written by Kevin Kelley of Anthropic for the Internet Engineering Task Force’s AI Preferences working group, and it is deliberately simple. You do not need a new file format, a new crawler name, or a paid tool. You add one group to the robots.txt file at the root of your site.
How to write the new rule
A site-wide opt-out looks like this:
User-Agent: __usage_ai-training Disallow: /
To keep part of the site open, add an exception, for example a press kit:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
The trick is that the token names a use, not a bot. Ordinary robots.txt rules name a crawler, like GPTBot or Google-Extended. This token names the activity, AI training, so one group covers all four companies at once. It also reuses the familiar Allow and Disallow path rules, so a section-level opt-out works the way site owners already expect.
There is one trap worth knowing. The familiar wildcard rule, User-Agent: *, does not express an AI training preference under this draft. Kelley, announcing the draft to the working group, said that was deliberate. A robots.txt file that does not use this mechanism expresses no preference at all. Silence is not an opt-out.
When each company actually starts honoring it
| Company | Start date | Notes |
|---|---|---|
| Anthropic | Already implementing | The draft’s author; earliest mover of the four |
| OpenAI | Coming months | No firm date given; existing GPTBot controls stay in place |
| Microsoft | February 1, 2027 | Separate grounding controls arrive alongside it |
| 2027 | Early 2027 per crawler documentation; no exact date yet |
Google’s handling has one useful detail. John Mueller, a Google search advocate, said a site that opts out with the shared token and does not mention Google-Extended will be treated as if it had blocked Google-Extended, Google’s own AI training control since 2023. Conversely, sites that explicitly allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks through grounding, the live retrieval step that feeds answers.
What the token does not do
This is where site owners need to read carefully, because the new line is narrower than it first sounds. First, it controls training only. Training, as defined in the working group’s vocabulary draft, means using content to modify the learned parameters of a generative AI model. It does not cover search indexing, and it does not cover grounding, where an AI tool reads a page in real time to answer a question.
Second, it is not a crawler block. The draft states plainly that the token “is not a directive for crawling behavior.” A bot can still visit, fetch, and cache a page that is opted out of training. If you want to stop crawling itself, you still need the old per-crawler rules.
Third, it is not retroactive. Updating robots.txt does not remove material already collected into datasets or already trained into model weights. OpenAI framed the mechanism as a coordination mechanism rather than a security control, and scrapers that ignore robots.txt will ignore this too.
Fourth, it is not yet a standard. The document is an individual Internet-Draft, dated October 30, 2026, with an expiry of May 3, 2027 unless renewed. The IETF datatracker notes that an individual draft has no formal standing and is not endorsed by the IETF. It binds only the companies that choose to follow it, and Kelley has warned that how Anthropic applies the signal may change as the work goes on.
Microsoft’s separate deadline is the one to diary
Microsoft is adding a second layer that the shared token does not cover: AI grounding controls for Bing, Copilot, and Microsoft 365, set page by page with a meta tag or HTTP header. More urgent for existing sites, Microsoft will stop recognizing its older NoArchive and NoCache tags as training and grounding signals after March 1, 2027. Any site that today relies on those tags to keep content out of Microsoft’s AI must migrate to the new controls before that date.
Microsoft says neither the new training preference nor the grounding preference changes eligibility for ordinary Bing search results, subject to normal crawling and indexing. That separation matters: opting out of training is not the same as disappearing from search.
Should you use it? A practical checklist
Start by deciding what you actually want to stop. If the goal is to keep content out of future training sets, add the shared group and keep your existing per-crawler rules in place until every company has implemented the token, which for Google and OpenAI means some time in 2027. If the goal is to stop a specific bot visiting at all, the token alone will not do it.
Next, check for conflicts. Search Engine Journal’s reporting notes that it is not yet clear what happens when a shared-line rule and a company-specific rule disagree, for example if a site blocks training generally but allows one company’s crawler. Until that is clarified, keep the two layers consistent rather than mixing allow and disallow across them.
Then, set the Microsoft deadline. Diary February 1, 2027 for Microsoft’s start date and March 1, 2027 for the NoArchive and NoCache cutoff, and audit any pages that rely on those tags now rather than in February.
Finally, weigh the tradeoff honestly. For a news site or blog chasing discovery in AI chat, a blanket opt-out trades away exactly the visibility that AI answers can bring. That is why this site keeps its content open to AI crawlers: the goal here is to be quoted and linked by AI assistants, not hidden from them. The new token is most useful for publishers with licensed, paywalled, or sensitive material where training use is the specific harm, not for sites that want to be found.
The real significance of October 30 may be less the token itself than the fact that four rival labs agreed on anything at all. Publishers have asked for one shared control for years, and they finally have a draft to point at. It is voluntary, staggered, and incomplete, but it is also the first time a single robots.txt line has meant the same thing to Google, Microsoft, OpenAI, and Anthropic. Site owners should use it with eyes open, keep their old controls until the new one is live everywhere, and remember that a preference is not a lock.
Sources: Internet-Draft draft-kelley-aipref-training-preference-signal-00, October 30, 2026; Search Engine Journal reporting, November 1, 2026; company announcements from Google, Microsoft, OpenAI, and Anthropic, October 30, 2026. For context on AI costs, see our October token pricing comparison and Mistral Large 4’s open-weights release.
