AIPREF robots.txt AI preferences: what to deploy before March 2027

Summary. There is still no ratified standard for telling AI systems how to use your content, so refusing AI training while keeping search in 2027 means vendor tokens plus one new shared rule. The IETF AI Preferences (aipref) working group's August 2026 milestones passed with neither specification sent to the IESG. Its vocabulary, -08 of 14 September 2026, defines train-ai, ai-use and search, says it does not reflect consensus, and expires on 18 March 2027. The attachment draft, -05 of 19 August 2026, expires on 20 February 2027. The change came on 30 October 2026: an individual draft from Anthropic's Kevin Kelley defines a robots.txt policy token, __usage_ai-training. Microsoft will honour it from 1 February 2027, Google plans support in early 2027, OpenAI aims for the coming months, and Anthropic has started implementing. Deploy now: keep search crawlers allowed, disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, Meta-ExternalAgent and CCBot if you refuse training, add a __usage_ai-training group, and if you rely on Bing's NoArchive or NoCache tags, migrate before 1 March 2027.

About this article. For SEO leads, web engineers and publishers who manage robots.txt. We read draft-ietf-aipref-vocab-08, draft-ietf-aipref-attach-05, draft-kelley-aipref-training-preference-signal-00 and RFC 9309, and checked status against the IETF Datatracker and crawler tokens against each operator's own documentation, on 1 November 2026. This is general information, not legal advice. Researched with AI assistance and reviewed by our editorial team.

Where AIPREF stands on 1 November 2026

The aipref working group sits in the IETF's Web and Internet Transport area, chaired by Mark Nottingham and Suresh Krishnan. Its charter covers a vocabulary, ways to attach preferences to content (including robots.txt and HTTP headers), and reconciling conflicting preferences. Technical enforcement, crawler authentication, registries and training audits are out of scope.

Both milestones were dated August 2026 and both slipped. On the Datatracker, the vocabulary's IESG state is still I-D Exists, with no document shepherd. Draft minutes of the 21 October 2026 interim record that -08 lacks consensus, and the group next meets at IETF 127 in San Francisco, on 16 and 19 November 2026.

The table comes from the Datatracker.

Expiry does not end the work. Internet-Drafts last at most six months and are normally replaced by a new revision, so treat today's text as a snapshot.

The process is less settled than the October headlines suggest. Nate Hake, a publisher who participates in the group, posted on 30 October that the work is unfinished at the IETF and that significant disputes remain, as reported by Search Engine Roundtable. On the list he has objected to the AI Use category's exception for use by users. Kelley himself offers his draft as input to the group's attachment work, not as a finished result.

What the AIPREF vocabulary defines

Version -08 defines three categories and a short text format.

• Ranking is not generative. The generative AI model definition excludes models used for classification, ranking or scoring, even when they generate a rationale. Search ranking systems sit outside train-ai on that reading.

• Search overrides the others. Allowing search permits processing inside a search application, including training models whose outputs are used only to rank and excerpt with links.

• The most restrictive preference wins. If any statement disallows a use, the result is disallowed. With no statement, every category is unknown, and the draft takes no position on what default applies.

• Preferences are not enforcement. The specification does not ensure preferences are followed, and agreements that explicitly permit a use likely apply anyway.

The exemplary format uses y or n, for example train-ai=n, search=y. Labels are case sensitive, unknown labels are ignored, and an open issue asks whether a URL a user pastes counts as directly provided.

Two ways to attach a preference

Content-Usage: the attachment draft

draft-ietf-aipref-attach-05, by Gary Illyes of Google and Martin Thomson of Mozilla, adds a new robots.txt rule and an HTTP response header that carry the vocabulary string.

            # robots.txt
User-Agent: *
Allow: /
Disallow: /never/
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

# HTTP response header
Content-Usage: train-ai=n
          

It is more expressive, but it adds new grammar to robots.txt, and Kelley reports that deployed parsers behave differently here. His 30 October message describes it as the draft the group put aside to progress the vocabulary. None of the 30 October vendor statements mentions Content-Usage, and we found no crawler operator committed to reading it.

The policy token: what vendors committed to

Kelley's draft takes the opposite approach: no new grammar. You write an ordinary robots.txt group whose User-Agent line names a policy token instead of a crawler.

            User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
          

That reads as a preference against AI training everywhere except /press-kit/, using RFC 9309's longest-match rules. The key rules:

• Wildcards never match. A User-Agent: * group is not read as matching the token, so a file without a __usage_ai-training group expresses no preference, even if it disallows everything for *.

• It is about use, not crawling. Crawlers still obey rules addressed to their own names, and a consumer may skip fetching pages it cannot use.

• Vendor tokens keep their meaning but do not count. A Google-Extended or GPTBot rule is not read as a shared preference.

• Specific beats general. Consumers may treat their own proprietary tokens as more specific than the shared one, and private agreements may override preferences.

• Timing follows retrieval. Preferences attach using the robots.txt in effect at fetch time, and the attachment draft says changes are not retroactive.

Who has committed to what

The table summarises the four statements of 30 October 2026.

OpenAI describes the signal as covering foundational generative AI training, and PPC Land notes that it does not say whether that is narrower than the vocabulary's definition. Google says that if a site opts out with the token and does not specify Google-Extended, it plans to read that as a Google-Extended disallow, and that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.

That coupling matters because Google-Extended is broader than training. Google's crawler page says it governs training of future Gemini models and grounding in Gemini Apps and on Vertex AI, does not affect inclusion in Google Search, and is not a ranking signal.

Microsoft separates the two uses. Its AI Grounding preference is page-level only, with Allow, Disallow and a new DisallowExtended that keeps content in Microsoft-operated experiences while excluding grounding it provides to third parties.

Verified AI crawler tokens

Every row comes from the operator's own documentation.

The robots.txt to deploy now

This file refuses AI training for the training tokens in the table, keeps search and AI-answer crawlers allowed, and adds the shared token for 2027. Replace the example disallow and sitemap with your own.

            # Shared AI-training preference (draft-kelley-aipref-training-preference-signal-00)
# Read by Microsoft from 1 Feb 2027; Google plans support in early 2027
User-Agent: __usage_ai-training
Disallow: /

# Vendor training controls: keep these until every vendor reads the shared token
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: Amazonbot
User-Agent: meta-externalagent
User-Agent: CCBot
Disallow: /

# Everyone else, including Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot and Applebot, follows this group
User-Agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml
          

Two deliberate choices. First, search crawlers get no groups of their own: under RFC 9309 a crawler that matches a named group ignores the * group, so a separate Googlebot group would silently drop your existing * disallows. Applebot follows Googlebot's rules if your file mentions Googlebot but not Applebot. Second, the training tokens share one group with several User-Agent lines, which RFC 9309 and the Kelley draft both allow.

On Cloudflare, Bot Preference Sync prepends its managed block to your file, and its documentation of 22 October 2026 shows no shared token, so keep your own group. Our note on the prepend gap explains how the blocks combine.

Variant: opt out of training but stay eligible for Gemini grounding. Add an explicit Google-Extended allow. Because Google-Extended covers training and grounding together, this also allows Gemini training.

            User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /
          

Variant: allow training on public docs only. Use path rules inside the policy-token group, for example Disallow: / plus Allow: /docs/. Google has not yet said how it will map path-level rules.

Page-level signals for Bing. Microsoft supports per-page preferences. A page-level training preference overrides robots.txt for that page, and grounding is page-level only.

            <meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">
          

The HTTP equivalent is X-Robots-Tag: Bingbot: ai-training:disallow. For Apple, the nosnippet directive keeps content out of AI-generated answers in Siri and Search.

Eight traps to avoid

1. The wildcard group says nothing about training. Under the Kelley draft, a blanket User-agent: * disallow is not a training preference. Add the policy-token group.

1. Google-Extended couples training and grounding. On Google's stated plan, a shared-token disallow without a Google-Extended rule also removes you from grounding in Gemini Apps.

1. Blocking mixed-use crawlers removes search. Cloudflare's Block setting now stops Googlebot, Bingbot and Applebot; its Disallow AI Training setting does not.

1. User-triggered fetchers may ignore robots.txt. OpenAI, Perplexity and Meta all say so for their user-action agents.

1. Changes are not instant or retroactive. Crawlers may cache robots.txt for up to 24 hours, and preferences attach at retrieval time. Microsoft must recrawl a page to see new settings, and IndexNow does not guarantee a recrawl date.

1. Bing's old tags are being retired. After 1 March 2027, NoArchive and NoCache stop counting as AI preferences, and once Microsoft detects any new AI preference on a site it disregards remaining tags. To keep withholding grounding, publish page-level ai-grounding preferences and record the temporary preference in Bing Webmaster Tools, open from 15 December 2026 only to sites using those tags on that date.

1. User agents can be spoofed. Verify with the IP lists operators publish, or with reverse DNS where offered.

1. llms.txt is not a control. Google says you do not need new machine-readable files, AI text files or markup to appear in its AI features, and neither AIPREF working group draft uses llms.txt. Our llms.txt analysis has more.

Timeline for 2026 and 2027

The dates come from the Datatracker, the IETF mail archive, Microsoft, Google and the CMA.

EU, UK and India notes

EU. Google says the shared token is consistent with the EU Code of Practice for general-purpose AI model providers, which it describes as recognising robots.txt as a primary way to express machine-readable preferences.

UK. The CMA's publisher conduct requirement adds controls for Google's AI Overviews and AI Mode: directory-level on 3 December 2026 and page-level on 3 March 2027. Those sit with the Search generative AI control and snippet controls, not the shared token. See our guide to CMA AI Overviews metrics.

India. A DPIIT committee's working paper of December 2025 recommended by majority a mandatory blanket licence with statutory royalties, under which rights holders could not withhold works from training. If adopted, a robots.txt opt-out would not stop training under Indian law. No revised paper has been reported. On 24 July 2026 the Delhi High Court refused ANI an interim injunction against OpenAI on a preliminary fair dealing view, and the appeal is reported as listed for 8 December 2026. Our AI crawler monetisation comparison covers the licensing side.

Implementation checklist

1. Export 30 days of server or CDN logs and list every AI user agent, verified by IP range.

1. Ship the robots.txt above on every host and subdomain, keeping your existing * rules.

1. Decide whether Gemini grounding matters enough to allow Google-Extended explicitly.

1. If you use NoArchive or NoCache, add Bingbot page-level preferences by 1 March 2027 and record the temporary grounding preference from 15 December 2026.

1. Test the file with a parser. Blank lines do not end a group under RFC 9309, so check the policy token is not absorbed into an earlier group.

1. Re-check the Datatracker after IETF 127 and before 18 March 2027 for vocab-09, a new attachment draft or a new Kelley revision.

How eCorpIT can help

eCorpIT is a Gurugram-based technology organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with senior-led engineering teams working across AWS, Microsoft and Google platforms. Our performance marketing services team can audit AI crawler traffic, set robots.txt and CDN rules that keep search visibility while refusing training, and plan migrations such as Bing's NoArchive deadline. Talk to us at /contact-us/.

Last updated: 1 November 2026.

References

1. AI Preferences (aipref) working group charter and milestones — IETF Datatracker

1. draft-ietf-aipref-vocab-08, A Vocabulary For Expressing AI Usage Preferences, 14 September 2026 — IETF Datatracker

1. draft-ietf-aipref-attach-05, Associating AI Usage Preferences with Content in HTTP, 19 August 2026 — IETF Datatracker

1. draft-kelley-aipref-training-preference-signal-00, 30 October 2026 — IETF Datatracker

1. Your content, your choice: clearer preferences for training and grounding, 30 October 2026 — Microsoft Bing Blog

1. Google's common crawlers — Google Crawling Infrastructure

1. Making it easier for websites to express AI training preferences, 30 October 2026 — OpenAI

1. New draft: draft-kelley-aipref-training-preference-signal-00, Kevin Kelley, 30 October 2026 — IETF mail archive

1. AI Preferences WG interim meeting minutes, 21 October 2026 — aipref working group

1. aipref sessions scheduled for IETF 127, 23 October 2026 — IETF mail archive

1. Google adopts new AI training control, 30 October 2026 — Search Engine Roundtable

1. Why are we protecting non-consensual AI user usage?, Nate Hake, 21 October 2026 — IETF mail archive

1. Supporting a shared AI training control for websites, 30 October 2026 — Google Search Central Blog

1. Google, Microsoft, OpenAI and Anthropic back one AI training opt-out, 31 October 2026 — PPC Land

1. Overview of OpenAI crawlers — OpenAI

1. Does Anthropic crawl data from the web, and how can site owners block the crawler?, 7 April 2026 — Anthropic

1. Perplexity crawlers — Perplexity

1. About Applebot — Apple Support

1. Amazon web search crawlers, Amazonbot — Amazon Developer

1. Meta web crawlers — Meta for Developers

1. CCBot — Common Crawl

1. RFC 9309, Robots Exclusion Protocol, September 2022 — RFC Editor

1. Bot Preference Sync robots.txt setting, last updated 22 October 2026 — Cloudflare Docs

1. Have it both ways: stay discoverable in search while disallowing AI training, 15 September 2026 — Cloudflare Blog

1. AI features and your website — Google Search Central

1. Publisher Conduct Requirement Notice, 3 June 2026 — Competition and Markets Authority

1. Search generative AI control — Search Console Help

1. Working Paper on Generative AI and Copyright, Part 1, December 2025 — DPIIT

1. India's AI copyright committee has yet to publish Part 2 of its working paper, 6 October 2026 — Digital Watch Observatory