AI usage preferences - the IETF draft that replaces a tangle of vendor rules
Tests for the Content-Usage header or robots.txt rule from the IETF aipref working group draft.
What this signal tests
We check whether your site declares AI usage preferences using the IETF aipref working group's draft format: either a `Content-Usage` HTTP response header, or a `Content-Usage:` line in robots.txt. The vocabulary uses simple tokens like `train-ai=y|n` and `search=y|n` to say, per resource, whether the page may be used for AI training, AI search, or both. The format follows RFC 9651 structured fields.
Why it matters for your visibility in AI
Today, blocking AI training requires you to maintain a growing list of vendor-specific user-agent rules: GPTBot, ClaudeBot, Google-Extended, CCBot, anthropic-ai, PerplexityBot, Applebot-Extended, and a dozen more. Every new model means a new rule. The IETF aipref draft (scheduled for IESG submission in August 2026) consolidates all of that into one cross-vendor vocabulary. Adopt it now and your policy survives the next wave of crawler launches without updates. The forward-looking benefit is meaningful: when aipref becomes the standard, sites that already use it will be honoured by every compliant crawler from day one, while sites still relying on per-vendor robots rules will have gaps. Cloudflare, Microsoft, and several major publishers are participating in the working group, so vendor uptake is likely. Until the draft stabilises, the smart move is to emit both the new header and your existing robots rules in parallel.
Pass criteria at a glance
| Criterion | Passes when |
|---|---|
| Either header or robots.txt rule declares train-ai and/or search with valid values. |
How we test it
We HEAD a sample of your URLs and look for the `Content-Usage` response header, validating it parses as an RFC 9651 structured field with the aipref vocabulary tokens (`train-ai`, `search`, with values `y`, `n`, or a policy URL). We also fetch /robots.txt and look for `Content-Usage:` lines. We pass the signal if either channel declares at least one valid aipref directive. Mixed channels are fine. The draft is at https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/.
Show technical detection method
HEAD URLs and check Content-Usage header; parse robots.txt for Content-Usage: lines. Validate against RFC 9651 structured fields and aipref vocabulary.
If your site fails: how to fix it
- Read the aipref working group drafts at https://datatracker.ietf.org/wg/aipref/about/ - particularly the vocabulary draft and the attach draft - to understand the current token set.
- Emit a response header like `Content-Usage: train-ai=n, search=y` from your origin or CDN if you want to allow AI search but block training, for example.
- Mirror the same policy in /robots.txt once the attach draft stabilises, using `Content-Usage:` lines per the spec.
- Keep your existing per-vendor robots rules (GPTBot, ClaudeBot, etc.) in parallel until aipref reaches RFC status. Belt and braces.
- Because this is still an IETF draft, have a developer who can read the spec implement it and revisit after each draft revision.
Quick facts
| Maturity | PROPOSED |
|---|---|
| Weight | medium |
| Category | Emerging Standards |
Primary sources
Related signals
Frequently asked questions
Is aipref a real standard or just a proposal?
It is an active IETF working group draft. The vocabulary and attachment drafts are well-developed and likely to become an RFC in 2026 or 2027. "Draft" in IETF land means "reasonably stable proposal undergoing review" - not "speculative."
Which AI crawlers honour Content-Usage today?
A small but growing number, including some Cloudflare-fronted bots. Most major crawlers still rely on robots.txt user-agent rules. Adoption will accelerate once the RFC publishes.
Should I drop my GPTBot / ClaudeBot rules in robots.txt?
Not yet. Until aipref is widely honoured, the per-vendor rules are still the operational mechanism. Run both. Once aipref is an RFC and major vendors confirm compliance, you can prune the legacy list.
Will fixing this still matter in two years?
Almost certainly yes - and more than it does today. A site that adopts aipref now will be properly honoured by the next generation of crawlers from day one, with no migration work.
Run your own scan
Run a free scan and see how your site grades across all 155 AI-readiness signals.