AI training: one robots.txt line to say no

Until last week, telling an AI company not to train on your site meant one line per crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, CCBot for Common Crawl, and another one every time a newcomer showed up. On October 30, 2026, four companies announced on the same day that they would all read the same token. Two lines are now enough. Which leaves the question of what they actually protect.

In short: an Internet-Draft filed with the IETF on October 30, 2026 by Kevin Kelley, an engineer at Anthropic, describes a single token to write in robots.txt — __usage_ai-training — to refuse having a site’s content used to train generative models. Google, Microsoft, OpenAI and Anthropic said the same day that they would honour it, on timelines ranging from right now to 2027. This is not a standard: it is an individual submission, not endorsed by the IETF, expiring on May 3, 2027. One trap went almost unnoticed: a site that already blocks every crawler with User-Agent: * has, under this text, expressed no preference at all. And the signal only helps whoever controls their own robots.txt.

What was announced on October 30: one token, four companies

It starts with a technical document: « Machine-Readable Training Preference Signal », filed with the IETF on October 30, 2026 by Kevin Kelley, of Anthropic. It proposes exactly one thing: a policy token, __usage_ai-training, to drop into an ordinary robots.txt group. That same day, Google published a post by John Mueller, Microsoft a post by Krishna Madhavan, Principal Product Manager at Bing, and OpenAI an announcement page.

The vocabulary itself comes from the IETF’s AIPREF working group, whose reference document defines AI training as « using an asset to modify the learned parameters of a generative AI model », and excludes models used exclusively for non-generative tasks — ranking, classification, scoring. Search ranking therefore falls outside the token’s scope. That is the thing everyone confuses.

What the line looks like, and what it actually says

The example in the draft is three lines long:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Translation: no training on the site, except for whatever sits under /press-kit/.

The mechanism changes nothing about the existing syntax. The draft spells it out: « This document introduces no new directive names and no new grammar » — the path language, longest-match precedence and group structure defined by RFC 9309 all stay the same, and the token is matched case-insensitively.

Now the detail almost nobody mentions. The draft states that wildcard groups, User-Agent: *, « MUST NOT be interpreted as matching the policy token », and that without the token, no preference has been expressed. In practice: a site that already blocks every crawler on earth with a generic Disallow: / has, under this text, said nothing about training. If your reaction is « my site is already closed, I’m fine », go and read your file again.

Crawling, training, grounding: three different things

Crawling is a bot downloading your page. Training is using that page to modify a model’s weights. Grounding is fetching your page at the moment someone asks a question, answering from it and, usually, citing it.

The three are independent. Refusing training does not remove your site from search results, and does not necessarily stop an AI from summarising your page live. Google’s crawler documentation says so about its own token: « Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search ».

Microsoft goes further: its post separates two independent settings, AI Training and AI Grounding, replaces the old NoArchive and NoCache tags, and adds a DisallowExtended option that limits grounding to Microsoft-operated experiences (AI Mode, Copilot, Microsoft 365). Its position fits in one sentence: « Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work. »

Google made the opposite, blunter call. If a site opts out of training with the token without mentioning Google-Extended, Google « plans to interpret that as if the publisher had disallowed Google-Extended ». And Google-Extended also covers grounding in Gemini: websites that allow it « are eligible to appear in responses and receive clicks via grounding in Gemini Apps ». The shared token therefore also takes you out of Gemini citations, unless you write a separate Google-Extended rule — company-specific controls, the firm notes, « will continue to function alongside this new joint signal ».

Who honours what, and from when

The word « adopt » hides very different timelines.

Company	Enforcement	Detail
Anthropic	Started	« It’ll take us some time to add support across our services », Kevin Kelley told TheNextWeb
OpenAI	« In the coming months »	Existing GPTBot controls stay in force
Microsoft Bing	February 1, 2027	Temporary preferences in Webmaster Tools from December 15, 2026
Google	Early 2027	« […] in early 2027 », its crawler documentation says

Two more Microsoft dates matter: on March 1, 2027, NoArchive and NoCache stop counting for training; on September 30, 2027, the temporary preferences expire. Microsoft also writes plainly that these preferences establish no licensing or payment arrangement.

As for the document’s status, do not overrate it. Its page on the IETF datatracker is unambiguous: individual submission, « is not endorsed by the IETF », « has no formal standing in the IETF standards process », expiring on May 3, 2027. Not a standard: a convention four companies have promised to follow.

The European foot in the door: the AI Act and Article 4(3)

Why now? Google ties its decision to the Code of Practice for General-Purpose AI Models under the AI Act, whose final version is dated July 10, 2025. Its Copyright chapter, Measure 1.3, commits signatories to employ web-crawlers « that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt) », as specified in RFC 9309 — a measure anchored in the machine-readable rights reservations of Article 4(3) of Directive (EU) 2019/790.

Translation: in Europe, a plain text file invented in 1994 and only standardised in 2022 has become the legal vehicle through which a publisher reserves their rights. Which explains the sudden rush to agree on a shared vocabulary.

If you publish online: what you can do, and where you can do nothing

The signal only works for whoever controls the /robots.txt file of their domain. And only that one: RFC 9309 requires the rules to be accessible « in a file named “/robots.txt” (all lowercase) in the top-level path of the service ».

You control: your personal site on a domain you own, a project deployed on Vercel, Netlify or Cloudflare Pages, a self-hosted blog. File at the root, group added, done.

Mind the GitHub Pages trap. If your project is published at username.github.io/my-project/, the robots.txt in your repo is never read: only username.github.io/robots.txt counts. With a custom domain, your project becomes the root of its own host and its file applies.

You control nothing: LinkedIn profile, Medium article, public Notion page, shared Google Doc, public GitHub repository. There, the platform decides. If your portfolio matters to you, host it yourself.

What this does not fix

It is not retroactive. The token says nothing about models already trained. What has been absorbed stays absorbed.

There is no enforcement mechanism. PPC Land notes that OpenAI calls its own participation a « voluntary effort », and cites TollBit research according to which roughly 15% of AI page fetchers in Europe reached URLs sites had disallowed. The same article points out that the draft’s acknowledgements table credits 46 contributors — News Corp, Financial Times, BBC, the Electronic Frontier Foundation, the Internet Archive — but that Meta and Amazon, present in that table, have made no public commitment.

The whole logic is contested. Researcher Courtney Radsch wrote on November 2 that « an opt-out approach flips copyright on its head and puts the burden on right holders – including independent content creators and under-resourced news organizations ». Citing Cloudflare data, she notes that fewer than 40% of the world’s most visited domains even publish a robots.txt, and warns about the companies concerned writing the rule themselves.

And your dissertation? Checked on November 3, 2026: the robots.txt of theses.fr, the French national thesis portal, contains only a User-agent: * group with a Crawl-delay: 5, a sitemap and a list of disallowed paths. No mention of GPTBot, ClaudeBot, Google-Extended, CCBot or the new token — under the draft, no training preference is therefore expressed for the French theses indexed there. If this worries you, it gets settled at deposit time, through your institution’s dissemination rules.

Our reading

This is ergonomic progress, not protection. Going from one rule per crawler to a single rule removes a genuine chore. But a voluntary convention, with no sanction, carried by a document the IETF does not endorse, remains a promise. The European code of practice gives it some weight; that is the only serious lever.

The real news is pedagogical. October 30 made public a distinction most people had never heard of: crawling, training and grounding are three different things, and the four companies do not treat them the same way. Microsoft separates them, Google links them. The same file will produce different effects depending on the engine.

Concretely, for you. If you have a portfolio, a technical blog or a project deployed on your own domain, add the group today: three lines, no risk to your search ranking, and you are ready for February 2027. If you want to stay citable by assistants while refusing training, plan a separate Google-Extended rule. And if you mostly publish on platforms, accept it: the file is not yours.

FAQ

Will refusing training make my site drop in Google?

No. Google’s documentation states that Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal. At Google, however, the shared token counts as disallowing Google-Extended, which takes you out of grounding in Gemini.

My robots.txt already blocks every crawler. Is that enough?

No, and this is the main trap. The draft states that a wildcard User-Agent: * group does not match the policy token, and that without the token, no preference has been expressed. You need a dedicated User-Agent: __usage_ai-training group.

Does this protect my dissertation deposited at my university?

Not by itself. The signal only applies to sites whose robots.txt you control. For an institutional repository, what counts is your institution’s dissemination rules and the deposit agreement you sign.

Is this text an international standard?

Not yet. It is an Internet-Draft submitted as an individual contribution: the IETF datatracker states that it is not endorsed by the IETF, that it has no formal standing in its standards process, and that it expires on May 3, 2027.

Further reading

• Deploying your first site: where to host a project, and when you take back control of your robots.txt.
• ChatGPT’s invisible watermark and the EU: the other European push on traceability.
• AI and academic integrity at university: what institutions expect from you.
• The theses-hal connector: search French theses and HAL documents.