One robots.txt line opts you out of AI training. Read the catch.

Until now, opting out of AI training meant keeping a list of crawler names, one rule per company, and hoping you didn't miss a new one. The new approach uses a single name in robots.txt, __usage_ai-training, defined in an Internet-Draft by Kevin Kelley of Anthropic, dated October 30.

Each company gave its own timeline:

• OpenAI: it will honor the signal and aims to implement it in the coming months. It calls this a voluntary effort, and its existing crawler controls stay in place.
• Google: it aims to support the control in 2027. Its crawler documentation says early 2027.
• Microsoft: its preferences take effect February 1, 2027, as it crawls your pages.
• Anthropic: the draft's author wrote to the IETF list that Anthropic has already started implementing support, and that rollout across its services will take time. That is the one claim here that comes from a mailing list and not a company page.

It's ordinary robots.txt. The draft's example opts a whole site out of training, then makes an exception for one folder:

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press-kit/

Two details are easy to miss. A User-Agent: * group doesn't count, so the wildcard rule you already have says nothing here: if no group names the token, you've expressed no preference. And this is a preference about use, not a lock on the door. The draft says it isn't a crawling directive, and it's an individual submission with no formal IETF standing yet.

This is the part to read twice. Google says that if you opt out with the shared token and don't name Google-Extended, it plans to read that as disallowing Google-Extended. And per Google's documentation, Google-Extended covers two things: training future Gemini models and grounding, which means using your content to inform answers in Gemini Apps.

So a line you added to stop training can also take you out of grounding in Gemini Apps. Google says sites that specifically allow Google-Extended stay eligible to appear and earn clicks there, but that same token covers training too. At Google, those two travel together for now. Regular Google Search stays out of this. Google says Google-Extended doesn't affect inclusion or ranking there.

The other companies draw the line differently. OpenAI's crawler docs treat GPTBot (training) and OAI-SearchBot (search) as independent settings, and sites that opt out of OAI-SearchBot won't show in ChatGPT search answers, though they can still appear as navigational links. Anthropic's docs split ClaudeBot (training) from Claude-User and Claude-SearchBot, and warn that blocking those two may reduce your visibility. Microsoft separates training from grounding too, with grounding set page by page.

Marketing insight: Blocking training and blocking visibility are not the same decision. For a business that wants customers to find it, read each company's fine print before you opt out of anything.

Don't count on this to undo past training. Anthropic's docs describe restricting ClaudeBot as a signal about a site's future materials. Microsoft says its settings don't guarantee inclusion, traffic or payment, and publishing a preference does nothing until it recrawls the page. It's also voluntary. OpenAI encourages other AI providers to honor it, which tells you the signal isn't universal.

• Read your own robots.txt. Type yoursite.com/robots.txt into a browser. Look for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended, and find out who added those rules and why. Nobody remembers. Somebody did it anyway.
• Check for accidental blocks. If you block a search crawler, you can drop out of those answers, whether or not anyone meant to.
• Decide by business model. If people pay to read your content, opting out of training may be worth it. If your content exists to bring in customers, being found usually matters more, and you may have no reason to add this line.
• Mark one Bing date if it applies. If your site uses NoArchive or NoCache tags, Microsoft stops reading them as training or grounding preferences after March 1, 2027. Publish your new preferences before then.

A file most owners have never opened now helps decide whether AI tools can use your site when a customer asks who to hire. Nobody needs to rush into robots.txt tonight. Most of this isn't live yet. Your robots.txt does need an owner, though, and that person should know what each line costs before it goes in.

Not sure what your site is telling AI tools right now? Talk to our founders below. See you in the next edition.