OpenAI, Google, Microsoft and Anthropic back a shared robots.txt signal to let sites block AI training in one step

What happened

On October 30, OpenAI, Google, Microsoft and Anthropic each announced plans to honor a shared signal in robots.txt that lets website operators communicate their AI training preferences to multiple providers at once. The mechanism is an Internet-Draft submitted by Kevin Kelley, technical lead for standards at Anthropic, to the IETF's AI Preferences working group.

Anthropic has already begun supporting the new directive. OpenAI said it aims to implement the signal in the coming months. Google plans support in 2027, and Microsoft will begin honoring the new preferences from February 1, 2027.

How the shared signal works

The mechanism adds a new group in a site's robots.txt file using the policy token __usage_ai-training as the User-Agent line. A website that wants to block AI training across all participating providers adds Disallow: all. The token does not replace existing company-specific crawler controls, which continue to function alongside it.

Under the draft, a group that names the token and disallows a path expresses a preference against training on that content. A robots.txt file with no __usage_ai-training group expresses no preference. Wildcard groups (User-Agent: *) must not be interpreted as matching the token.

How each company plans to implement it

Anthropic has already started development to support the feature, according to Kelley's IETF mailing list message. OpenAI says its existing crawler controls such as GPTBot will continue to apply alongside the shared signal, which it plans to implement in the coming months.

Google says that when a site sets the shared opt-out without explicitly specifying Google-Extended, it plans to treat it as a block on Google-Extended, the company's existing AI training control. Sites that allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks through grounding. Google aims to support the signal in 2027.

Microsoft will begin honoring the new preferences on February 1, 2027. It is also introducing a separate page-level AI grounding preference that lets sites control whether their content can inform AI-generated answers. The company is replacing its NoArchive and NoCache tags, which will stop being recognized as training or grounding options after March 1, 2027.

What the signal does not do

The shared signal covers only generative AI training, not search indexing, AI grounding, or user-initiated fetches. A site can block AI training while remaining fully discoverable in search. The draft explicitly states that the token expresses a preference about content use, not an order about crawling.

The signal cannot undo past scraping. Any content already collected by AI companies before a site adds the token remains in their existing training data and model weights. The token also does not create any payment or licensing arrangement, and private agreements between a site and an AI company can override it.

Why it matters for publishers

Before this announcement, a publisher that wanted to block AI training had to identify and write separate rules for each company's crawler: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and multiple crawlers for Microsoft. When a new crawler appeared, the publisher had to update the file again.

The shared signal collapses that work into one entry. For small publishers and individual website operators who may not have the technical resources to track every AI crawler, the reduction in maintenance burden is significant. However, the signal remains voluntary and depends on each company enforcing it.

The standards process and limitations

The underlying document is an individual Internet-Draft submitted to the IETF's AIPREF working group. It is not an IETF consensus document and has no formal standing in the standards process until the working group adopts it.

Meta signaled support for a common signal in principle but did not commit to honoring the specific token. Andy Stone, Meta's head of communications, called it an important step on X without saying Meta would join. The signal's effectiveness depends on how many AI companies choose to honor it, and the earliest firm implementation date among the committed four is February 1, 2027.

Caveats

None of the four companies crawlers currently honors the new signal. Only Anthropic has begun development. OpenAI, Google and Microsoft have made commitments with varying timelines. The IETF draft remains an individual submission, not a standard. The signal is voluntary and does not address AI grounding, which is a separate concern for many publishers. The timeline gap between commitment and implementation leaves publishers in the same position for months.