🔍 Read the full analysis: AI Training Opt-Out: How To Set It On Supported Services on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a website’s content for generative AI training. The signal is voluntary, does not block crawling or cover AI answers from live web content, and companies plan to adopt it on different schedules.
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared website preference asking them not to use content for generative AI training. The proposed robots.txt rule gives publishers one signal to use across participating services, but it is a voluntary request, not a technical block, and the companies have set different implementation timelines.
The signal is described in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published October 30 by Kevin Kelley of Anthropic. Website operators who want to express the preference can add this group to their robots.txt file:
User-Agent: __usage_ai-training
Disallow: /
The rule expresses a preference against using the site’s content to train generative AI models. It does not block crawlers from accessing the site, remove material from models already trained, or determine whether a publisher receives payment.
The companies’ schedules differ. Anthropic says it has started implementing support, while noting that adding support across its services will take time. OpenAI says it aims to implement the signal in the coming months. Microsoft says it will honor the preference from February 1, 2027, as it crawls sites. Google aims to support it in 2027; its crawler documentation specifies early 2027.
Microsoft is also introducing a separate page-level preference for AI Grounding—using web content to inform answers in Bing, Copilot and Microsoft 365. Its announcement says both preferences take effect February 1, 2027. Microsoft’s existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027.
Existing crawler controls, including GPTBot, Google-Extended and ClaudeBot, remain in place. Google says that if a site uses the new signal without a separate Google-Extended rule, it plans to treat the site as having disallowed Google-Extended. Microsoft says its preferences do not change eligibility for Bing search results, and Google says Google-Extended does not affect Google Search.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
Until now, publishers seeking to express training preferences have generally had to manage company-specific crawler rules. A common token could reduce that work for sites dealing with several AI providers and make the preference easier for participating companies to recognize. That is a practical change, especially for publishers whose material may be collected at scale.
But the signal does not provide the same control as a technical access restriction. Robots.txt communicates a request; it does not enforce compliance against a crawler that chooses not to follow it. The announced support also does not settle broader questions about how web material is used, whether publishers should be compensated, or whether systems can answer questions using current web pages.
Those limits matter for site owners deciding how to configure access. Training, search indexing and the use of pages to ground AI answers are separate activities. A site’s preference for one does not automatically settle its choices about the others. Microsoft’s separate grounding control makes that distinction explicit, while the shared token described in the draft covers training.
Products Worth Considering
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
LG UltraGear 45GX900A-B 45-Inch Curved OLED Gaming Monitor
Logitech MX Keypad Programmable Macro Pad, Graphite
As an affiliate, we earn on qualifying purchases.
From Individual Rules to One Signal
The proposal comes through the AI Preferences working group at the Internet Engineering Task Force, which develops internet standards. The draft is an individual Internet-Draft, not an adopted standard. At a working group meeting on October 21, one chair said, “We haven’t declared consensus on anything yet.” The draft itself has no formal standing in the IETF standards process.
Company announcements refer to the same draft, but their commitments are not identical. Meta expressed support for publishers stating preferences, and its communications chief called a common signal an important step. The statement did not say whether Meta will honor the token. The source material also reports no announcement from Apple, Amazon, xAI or Perplexity; that is not proof those companies will not adopt it later.
For publishers, the practical approach described in the announcements is to add the shared rule if they want to opt out of training, while retaining existing company-specific rules until each relevant service has implemented the new signal. Sites using Microsoft’s NoArchive or NoCache tags should review the company’s stated March 1, 2027 changeover date and consider its separate grounding preference if that use is also a concern.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Still Vary
Whether every major AI company will honor the rule remains unresolved. Meta has voiced support without making an implementation commitment, and the source material identifies no public adoption statement from several other companies. The list of participants could change as the proposal and company plans develop.
The timing and coverage of implementation also remain uneven. Anthropic says rollout has begun but may take time across its services; OpenAI has offered a general target of the coming months; Google’s target is 2027; and Microsoft has named February 1, 2027. The announcements do not establish that every service operated by each company will recognize the preference on the same date.
The proposal is also still a draft rather than an approved internet standard. How it will develop through the IETF process, whether additional companies will adopt it, and how consistently the request will be honored are not yet settled.
Publisher Actions Before Rollout
Website operators can add the proposed user-agent group to robots.txt now if they want to express a preference against training, but should treat it as one part of their crawler policy, not a guarantee that content cannot be accessed or used. They can keep existing provider-specific rules while waiting for each service to implement the shared signal.
Publishers using Microsoft’s NoArchive or NoCache tags should check their settings ahead of the announced March 1, 2027 change to how those tags count for training and grounding preferences. Sites that also want to limit the use of pages in AI-generated answers should review controls for that use separately; the shared training signal does not cover it.
The next developments to watch are company-specific implementation notices and the IETF working group’s treatment of the draft. Until those are clearer, site owners will need to distinguish between a stated preference, a company’s actual support for it and any separate controls for search or AI answers.
Key Questions
What rule can a website add to opt out of AI training?
The proposed robots.txt group is User-Agent: __usage_ai-training followed by Disallow: /. It expresses a preference against using the site’s content for generative AI training.
Does the rule stop AI companies from crawling a website?
No. The signal communicates a training preference; it does not technically block crawling. Its effect depends on whether a company supports and honors it.
Does opting out of training stop AI answers from using a site?
Not under this signal. The shared rule covers training, not AI answers based on live web content. Microsoft announced a separate AI Grounding preference for that use.
Should publishers remove existing crawler rules?
The source material says existing controls remain applicable. Publishers may keep company-specific rules until the relevant services have implemented the shared preference, and should check each company’s timing and instructions.
Will every AI company follow the shared signal?
That is not confirmed. OpenAI, Google, Microsoft and Anthropic announced support on differing timelines. Meta expressed support for publisher preferences but did not commit to honoring the token.
Source: ThorstenMeyerAI.com
