OpenAI, Google and Anthropic back one robots.txt signal for opting out of AI training

What was announced

OpenAI said on 30 October 2026 that it will honor a common signal in robots.txt that lets websites state whether their content may be used for foundational generative AI training. The company says the effort is voluntary and lets publishers send the same preference to several participating AI providers at once, instead of writing separate rules for each provider's crawler. OpenAI says it is joining other participants in the IETF AI Preferences (aipref) working group in the announcement, and that its existing crawler controls will continue to apply.

The signal is defined in an Internet-Draft submitted the same day by Kevin Kelley of Anthropic. In his post to the working group, Kelley wrote that Anthropic has already started implementing support and that rollout across its services will take time. Google added a note to its crawler documentation saying it intends to support the shared signal in early 2027. OpenAI gives no date beyond the coming months.

How the signal works

A site operator writes an ordinary robots.txt group whose User-Agent line names the policy token __usage_ai-training. Disallow: / followed by Allow: /press-kit/ expresses a preference against AI training for the whole site except the press kit. The draft adds no new directives or grammar, so existing parsers that do not know the token ignore it like any unknown user agent. Wildcard groups (User-Agent: *) never match the token, which means no current robots.txt file expresses a training preference until its owner adds the group.

The token takes its meaning from the working group's vocabulary draft: AI training is using an asset to change the learned parameters of a generative AI model. Models used only for classification, ranking or scoring are outside that definition. The draft also says the token is a statement about use, not a crawling rule. Rules for a crawler's own name, such as GPTBot or Google-Extended, keep their current meaning.

Why it matters

Until now a publisher that wanted to keep its pages out of model training had to list every provider's crawler or control token separately, and new providers kept adding names. One shared token reduces that work and leaves one plain record of what the site asked for. It is still a preference: the draft and OpenAI's post describe a voluntary signal, not an enforcement mechanism, and it binds only the companies that choose to honor it. The draft is an individual submission with no formal IETF standing yet. Its acknowledgments list contributors from Microsoft, Meta, Amazon Web Services and Cloudflare, but those companies had not published their own commitments at the time of writing. Related disputes over training data are collected on the AI Policy hub.

What to watch next

Kelley says he will propose the mechanism as input to the working group's attachment draft, and aipref sessions are scheduled at IETF 127. The open questions are which other AI developers commit to the token, how providers treat content collected before a site adds it, and whether regulators such as the European Commission, which consulted on copyright and AI in September, treat it as a valid machine-readable opt-out.

Questions people ask

How do I opt my website out of AI training with the new signal?

Add a robots.txt group with the line User-Agent: __usage_ai-training followed by Disallow: /. Paths you mark with Allow express a preference in favor of training.

Does the signal block crawlers from my site?

No. The draft says it is a preference about how content is used, not a crawling rule. Existing rules for named crawlers such as GPTBot still control fetching.

When will OpenAI, Google and Anthropic honor it?

Anthropic says it has started implementing support, Google says early 2027, and OpenAI says in the coming months.

Sources

1. Making it easier for websites to express AI training preferences (OpenAI, 30 Oct 2026)
2. Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) (IETF Datatracker, 30 Oct 2026)
3. New draft: draft-kelley-aipref-training-preference-signal-00 (IETF ai-control mailing list, 30 Oct 2026)
4. Google's common crawlers (Google for Developers, 30 Oct 2026)
5. A Vocabulary For Expressing AI Usage Preferences (IETF Datatracker, 13 Sep 2026)