Google, OpenAI, Microsoft and Anthropic said on 30 October 2026 that they will honour one shared rule in robots.txt, the plain text file that tells crawlers what they may fetch from a website. With it, a business can ask for its pages to be kept out of generative AI training in a single entry instead of writing a rule for each AI company. Anthropic says it has started implementing support, Microsoft will honour it from 1 February 2027, Google aims to support it in 2027 and OpenAI says it aims to do so in the coming months.
What happened
OpenAI published "Making it easier for websites to express AI training preferences" on 30 October 2026. In it, OpenAI says it will honour a common signal in robots.txt that lets websites state whether their content may be used for foundational generative AI training. It calls this a voluntary effort, so a publisher can send the same preference to several participating providers at once.
The same day, Google's Search Central blog posted "Supporting a shared AI training control for websites", written by John Mueller, Google Search Advocate. Google says the control builds on Google-Extended, the AI training setting it has offered since 2023.
Microsoft followed on its Bing blog with a post by Krishna Madhavan, Principal Product Manager, Microsoft Bing, titled "Your content, your choice: clearer preferences for training and grounding". Microsoft is adopting the shared training preference and adding a second, separate setting for grounding, which it describes as using content to help inform AI-generated answers.
The signal itself is an Internet-Draft called "Machine-Readable Training Preference Signal", written by Kevin Kelley of Anthropic and published on 30 October 2026 for the AI Preferences (AIPREF) working group of the Internet Engineering Task Force (IETF), the body that develops internet standards. Announcing it on the group's mailing list, Kelley wrote that Anthropic has already started implementing support and that it complements Anthropic's existing crawler controls rather than replacing them.
It is still a proposal. The IETF's datatracker says it is an individual draft that the IETF has not endorsed and that has no formal standing in its standards process.
Key details
| Company | What it said | When |
|---|---|---|
| Anthropic | Wrote the draft and has started implementing support; it complements Anthropic's existing crawler controls | Work under way; Kelley wrote that support across its services will take time |
| An opt-out with the shared token will be read as a Google-Extended opt-out, unless the site names Google-Extended itself | Aims to support it in 2027 | |
| Microsoft | Honours the shared training preference, adds page-level training and grounding settings, replaces its NoArchive and NoCache preferences | From 1 February 2027, as it crawls |
| OpenAI | Will honour the signal; its existing robots.txt crawler controls stay in place | Aims to implement it in the coming months |
How the rule is written, as the draft describes it:
- A new "policy token" called __usage_ai-training goes in the User-Agent line of an ordinary robots.txt group. No new grammar is added to the robots.txt standard (RFC 9309).
- The draft's own example is three lines: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/. That asks for no AI training on the whole site except the press kit folder.
- A wildcard group (User-Agent: *) does not count. If the file has no group with the token, no preference has been expressed.
- The token is about how content is used, not a crawling instruction. To limit crawling, you still write rules for each crawler's own name.
- "AI training" means using content to change the learned parameters of a generative AI model. Training models that only classify, rank or score is not covered.
- Private agreements between a site and an AI company may override the preference.
Why it matters
Until now, OpenAI says, website operators had to identify each company's crawler and keep separate rules for every provider. One shared token is far easier for a small business to set and to audit.
The draft also says the preference that applies to content is the one in the robots.txt file in effect when that content was retrieved. A rule you add today speaks to future collection; it does not reach back to pages already gathered.
The bigger shift is that training and AI answers are being split into separate choices. Google says websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. Microsoft warns that choosing Disallow or DisallowExtended for grounding will significantly reduce a site's visibility in AI-generated experiences, while leaving its eligibility for Bing search results unchanged. Microsoft's DisallowExtended option keeps grounding inside Microsoft's own products, such as Copilot and Microsoft 365, and excludes the grounding services it provides to third parties.
There is also a regulatory angle. Google says the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to state machine-readable preferences. We covered how EU AI Act transparency rules can reach Australian businesses in an earlier post.
Finally, it is a signal that participating companies have said they will respect, not a lock, and Microsoft says it sets up no licensing or payment arrangement.
What this means for businesses
Most businesses publish pages they want customers to find, and some also publish material they would rather keep out of a model, such as paid guides. An export business such as Truly Ceylon, for example, publishes buyer guides and product pages written for importers to find. The shared token lets you treat each kind of page differently, folder by folder, without blocking search engines.
A short checklist:
- Find out who can edit your robots.txt file: your web developer, your hosting provider or the website platform.
- Open your current file (add /robots.txt to your domain) and write down any rules for named AI crawlers or for Google-Extended. Google and OpenAI say their company-specific rules keep working alongside the new token.
- Decide on training and on AI answers separately. You might keep the public service pages open to both, and ask for no training on a resources folder.
- Do not rely on a User-Agent: * line to say no to training. The draft says wildcard groups do not count.
- If your site uses NoArchive or NoCache tags for Bing, note Microsoft's dates: temporary grounding settings open in Bing Webmaster Tools on 15 December 2026, the new preferences apply from 1 February 2027, the steps should be done before 1 March 2027, and temporary settings expire on 30 September 2027.
- Remember that robots.txt is not a privacy or security control. Google says it is not a way to keep a web page out of Google; for that you use noindex or a password. Anything confidential belongs behind a login.
- Record the decision in your website change log and in the brief for any rebuild. Our website redesign checklist covers the other files and settings that tend to disappear in a rebuild.
Bots that ignore preferences are a different problem: for agents probing forms and logins, start with our notes on lessons for business websites after an AI agent breach and on the common weak spots AI test models got past.
If your site was built some time ago and nobody is sure what its robots.txt says, our website team can review it alongside the rest of the site, and our IT consulting work covers wider decisions about how your business uses AI tools. Ask us for a free first consultation; we agree a written quote before any work starts.
Frequently asked questions
Does the new robots.txt rule keep my website out of AI training today?
Not yet for most providers. Anthropic says it has started implementing support, Microsoft will honour it from 1 February 2027, Google aims to support it in 2027 and OpenAI says the coming months. Google and OpenAI say their existing company-specific rules keep working in the meantime.
Will asking for no AI training remove my business from Bing or Google search?
Microsoft says the new preferences do not change eligibility for traditional Bing search results. Google says it will read the shared opt-out as a Google-Extended opt-out, which is its separate AI training setting, unless your file names Google-Extended itself.
Does a User-Agent: * rule already cover AI training?
No. The draft says wildcard groups must not be read as matching the AI training token, so a file without a User-Agent: __usage_ai-training group expresses no preference about training at all.
Is this a law or an official internet standard?
Neither, for now. It is an individual Internet-Draft that the IETF has not endorsed, and companies follow it voluntarily. Google says it is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act.
Will AI companies pay my business for using its content?
Not through this signal. Microsoft says the preferences do not establish licensing or payment arrangements, and that neither allowing nor blocking promises inclusion, traffic or payment.
Key takeaways
- Google, OpenAI, Microsoft and Anthropic back one robots.txt token, __usage_ai-training, announced on 30 October 2026.
- Support arrives in stages: Anthropic has started, Microsoft from 1 February 2027, Google in 2027, OpenAI in the coming months.
- Training and AI answers are separate choices; Microsoft says blocking grounding reduces visibility in AI answers.
- Wildcard rules do not count, and robots.txt is a preference, not a security or privacy control.
- Check who controls your robots.txt and record the decision before your next website change.