In this post 9 sections
On 30 October 2026 Google, Microsoft and OpenAI each said they plan to honor one shared robots.txt group, __usage_ai-training, to express an AI-training opt-out, but none of them honors it yet. Microsoft starts on 1 February 2027, Google aims for early 2027 and OpenAI says "in the coming months." What works today is naming specific crawlers in robots.txt, Google-Extended for training, and edge-level blocking. llms.txt as a visibility lever remains folklore. Current as of 13 November 2026.
Key Takeaways
- The shared token comes from an individual IETF Internet-Draft by Kevin Kelley of Anthropic, dated 30 October 2026, which the IETF does not endorse and which expires 3 May 2027 (IETF Datatracker).
- It covers training only. It is a preference signal, not a crawl directive, it does not undo past collection, and a
User-Agent: *group does not count as matching it. - Google's Gary Illyes says robots.txt is an access preference protocol: it controls which URLs crawlers may request, not what happens to the data afterward, and to block a specific AI crawler you must name its token (Search Engine Journal, 22 October).
- Cloudflare's 15 September changes added a "Disallow AI Training" setting and Bot Preference Sync, but a HasData crawl found Bot Preference Sync in none of 6,451 robots.txt files it collected.
- Google's own AI features documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews and AI Mode.
- Anthropic's support status is disputed: Engadget says Anthropic already supports the token; Search Engine Journal, citing Kelley's mailing-list post, says development has started and none of the three companies supports it yet. I trust the second account because it names its source.
Status of each control
| Control | What it does | Status as of 13 Nov 2026 | My verdict |
|---|---|---|---|
| Named user-agent rules (GPTBot, ClaudeBot, OAI-SearchBot and similar) | Tell compliant crawlers not to fetch | In use; no enforcement | Works for compliant bots; blocks nothing else |
| Google-Extended | Controls use for future Gemini training and for grounding in Gemini Apps; no effect on Search | Live | Verified, per Google's own description |
| Search Console control for AI Overviews and AI Mode | Excludes a site from those features | UK CMA order on 3 June; Google testing with a limited set of UK sites (SiliconANGLE) | Rollout unclear; confirm availability in your account |
__usage_ai-training shared token | One line for training preference | Announced, not shipped | Pending; harmless to prepare |
| Cloudflare Disallow AI Training and Bot Preference Sync | Block training crawlers while allowing search | Live from 15 Sept for new domains | Verified as a Cloudflare setting; downstream effect unclear |
| Cloudflare pay per crawl | Charge crawlers per fetch | Private beta, per Cloudflare's docs updated 14 Aug 2026 | Not available to most sites |
| llms.txt | A Markdown summary file for AI systems | Google says Search ignores it | Folklore as a visibility lever |
| IETF aipref vocabulary and attachment drafts | Standard vocabulary for AI preferences | Working group drafts, milestones set for 31 Aug 2026 | Unfinished |
What changed in the last two weeks
The shared training signal
Google's post by John Mueller, dated 30 October, says it plans to support a shared AI training control in robots.txt. If a site opts out with the shared token and does not set Google-Extended explicitly, Google would treat it as if Google-Extended were disallowed. Sites that explicitly allow Google-Extended can appear in Gemini Apps responses and receive clicks through grounding. Google aims for 2027 and says the proposal is consistent with the EU Code of Practice for general-purpose AI. Microsoft's separate announcement (see my post on Bing's new AI grounding controls) adopts the token for training and adds page-level grounding controls. Search Engine Journal's report adds that neither OpenAI's blog post nor its crawler documentation says how a conflict between the shared rule and a GPTBot rule would be resolved.
The syntax, per the draft, is an ordinary robots.txt group:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/A path that is disallowed expresses a preference against use. A path that is allowed expresses a preference in favor. With no group for the token, no preference is expressed. A webscraper.io write-up warns that Python's urllib.robotparser applies first-match, so put Allow lines first.
Enforcement versus intent
Robots.txt shows what you ask for, not what a bot does. HasData, a scraping API vendor, tested 10,894 domains in 2026 with two requests each, one as GPTBot and one as Chrome. Its index, reported by PPC Land, found 234 of 592 sites (39.5%) that disallow GPTBot still served it a live page. The author notes the GPTBot requests carried the official user agent string but were not verified OpenAI traffic, so this measures server behavior, not whether OpenAI's crawler honors robots.txt.
The same index found, on Cloudflare sites that dropped GPTBot bans in September, that 118 of 133 archived files had carried Cloudflare's managed block with a machine-readable no-training signal, and none carried it on 16 September. Of 21 affected sites tested live, 16 still blocked or challenged GPTBot. HasData's account conflicts with Cloudflare's: Cloudflare says almost no existing customers need to act, while HasData, citing Cloudflare's engineering post, says legacy users must review and confirm Bot Preference Sync. Check your own robots.txt after any Cloudflare change, as I advised in Cloudflare, Googlebot and default blocks.
What is verified
- Search and training are now separable at several vendors. Cloudflare's 15 September post says Applebot, Bingbot and Googlebot stay allowed for search under Disallow AI Training, while training-only crawlers from Amazon, Anthropic, Meta and OpenAI are blocked. It lists limits: no Disallow for agents yet, and Microsoft's robots.txt no-training support was then targeted for early 2027.
- Blocking a training bot does not necessarily block ChatGPT search. I cover the difference in OAI-SearchBot vs GPTBot. A small crawl of 1,069 top sites on DEV Community, collected 4 November, found 139 of 211 sites that block GPTBot still allow OAI-SearchBot. It is a one-author analysis labeled "Fully Autonomous" on DEV, so I treat it as a hint only.
- Blocking the search crawler has a visible cost. Muck Rack reports that among major publishers blocking Claude's search crawler, 81% received almost no Claude citations, versus 12% among those allowing it (Muck Rack). It is a correlation, but it is the best data I found.
- Google-Extended does not affect Search. Illyes states it has no crawler of its own and controls training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
- Google's limit controls for AI features are the snippet tools. Google's AI features documentation lists nosnippet, data-nosnippet, max-snippet and noindex, and says robots.txt for Googlebot controls crawl access to Search.
What is folklore or unproven
llms.txt. Google's AI features page says "You don't need to create new machine readable files, AI text files, or markup" to appear in these features. The same page does not mention llms.txt by name. Secondary coverage reports a 15 June 2026 changelog entry, "Clarifying guidance on llms.txt files," and a Security Boulevard piece by a GEO vendor's executive says it states the file neither helps nor hurts rankings; I did not open the changelog itself. Claims that other AI companies read it rest on vendor data I could not verify. For a measured view, see is llms.txt worth it in 2026, and if you want to add one anyway, a working llms.txt in ten minutes takes less time than arguing about it.
Content Signals as protection. Cloudflare's machine-readable signals declare a preference. Preference is not enforcement, and HasData found Cloudflare's own managed signal removed from files in September. I did not open the Content Signals specification page itself, so I make no claim about its syntax.
Pay per crawl as a revenue plan. Cloudflare's AI Crawl Control page labels it private beta, last updated 14 August 2026. For background on the idea, see pay-per-crawl web. Do not budget revenue from it.
The IETF vocabulary as a done deal. The aipref charter set 31 August 2026 milestones for submitting the vocabulary and attachment drafts to the IESG. The page does not say whether they were met. Illyes's co-authored Content-Usage draft expires 20 February 2027 unless replaced, and the "train-ai" label sits in a vocabulary draft that reflects no working-group consensus, per Search Engine Journal.
What this means for marketers and founders
- Decide by purpose, not by bot list. Separate three questions: do you want to appear in AI search, do you permit training, and do you allow agents. Name the crawlers for each in robots.txt.
- Keep search-side crawlers allowed unless you have a licensing reason. The Muck Rack figure is the cost evidence.
- Check logs, not just robots.txt. Confirm what your CDN returns to GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. The method is in AI bot log file analysis.
- Prepare the shared token, do not rely on it. Adding a
__usage_ai-traininggroup is harmless to legacy parsers per the webscraper.io write-up, but no provider honors it today. - Spend ten minutes on llms.txt at most. Put the saved time into the checks in Technical SEO checklist 2026 for AI crawlers.
What we do not know yet
- Whether OpenAI will honor the shared token over an existing GPTBot rule, and when.
- Whether Anthropic honors it today. Sources conflict, as noted above.
- Whether Google's control for AI Overviews and AI Mode has reached every account. The SiliconANGLE report describes a limited UK test with a global rollout planned, and I found no later primary confirmation.
FAQ
What is the __usage_ai-training robots.txt line?
It is a proposed robots.txt group that states whether content may be used to train AI models. It comes from an individual IETF Internet-Draft by Anthropic's Kevin Kelley, and Google, Microsoft and OpenAI said on 30 October 2026 that they plan to honor it.
Does Google honor the new AI training robots.txt token yet?
No. Google says it aims to support it in 2027 and will treat it as a disallow of Google-Extended if you do not set Google-Extended explicitly. Existing Google-Extended rules keep working.
Does Google-Extended stop my site from appearing in AI Overviews?
No. Google's Illyes says Google-Extended controls training for future Gemini models and grounding in Gemini Apps and on Vertex AI. It does not affect Search inclusion, and AI Overviews and AI Mode use separate controls.
Does llms.txt help me get cited by ChatGPT or Google AI Overviews?
Google says you do not need AI text files to appear in its AI features. I found no confirmation from OpenAI or Anthropic that their answer systems read it, and no independent study I could verify showing a citation benefit.
If I block GPTBot, will ChatGPT still be able to cite me?
Possibly. GPTBot is a training crawler, while OAI-SearchBot serves search. Blocking one does not automatically block the other, but check your CDN rules too.
Does robots.txt actually stop AI crawlers?
It stops crawlers that choose to comply. HasData's test found 39.5% of sites that disallow GPTBot still served it a live page, which means robots.txt alone is not enforcement.
Is Cloudflare pay per crawl available to everyone?
Not as far as Cloudflare's documentation shows. It labels pay per crawl private beta.
Want help with this?
I'm Younus Fardeen, an organic growth, SEO and AEO strategist with 4+ years of marketing experience across edtech and startup teams. If you want help auditing which AI crawlers can reach your site and deciding what to allow, see my work and reach me through the contact form at younusfardeen.com.
Found this useful? Pass it on.