Website owners currently encounter different crawler names and controls when deciding whether public pages can be used to train AI systems. On October 30, 2026, Google announced plans to support a shared policy token in robots.txt, based on work from the IETF AIPREF working group. The proposal could make some preferences easier to express across participating providers, but the announcement describes future support rather than a ready-to-deploy universal setting.
What Google announced
Google says publishers will be able to use a shared policy token in robots.txt to express broad preferences across AI providers that participate. The company says the work builds on its existing Google-Extended control and the IETF's work on AI preferences. Google described how it plans to interpret the shared signal in relation to Google-Extended when a publisher does not separately specify that token.
The important qualifier is participation. A shared token can only guide services that recognize and honor it. Provider support, token syntax, policy meaning, and rollout timing need to be confirmed from implementation documentation before a site adds it.
Do not treat training, crawling, and search as the same choice
Robots.txt rules are crawler instructions, and their effects depend on the crawler and the policy it follows. A preference about model training is not automatically a request to block a search crawler, remove a URL from search, or prevent every form of content use. Google says its Search crawler controls remain the way to manage access for Google Search.
Keep each decision explicit: whether a crawler may fetch a page, whether a page can appear in search, whether snippets can use selected text, and whether content may be used for model training. One directive should not be assumed to cover all of those purposes across all providers.
What site owners can do now
- Inventory existing robots.txt rules and provider-specific controls such as Google-Extended.
- Decide which content is public, which crawlers should fetch it, and what your training preferences are.
- Monitor the IETF AIPREF work and each provider's official documentation for final syntax and behavior.
- Test any new directive on a limited site section and verify the actual robots.txt response.
- Keep private or customer-only material behind access controls; robots.txt is not authentication.
- Record the change and the date so later crawler behavior can be compared against a known configuration.
Do not copy a proposed token from an announcement
Premature directives can have broader effects than intended, especially if they are mistaken for search-crawler access rules or copied before the syntax and provider behavior are final. Wait for published implementation instructions. If a provider has a separate, documented control today, assess it on its own terms rather than assuming the shared signal already replaces it.
A shared policy could reduce repeated configuration when providers adopt it. It cannot by itself prove that a site will be cited, excluded, or represented accurately in generated answers. Keep measuring actual crawl activity and sampled AI answers separately from policy settings.
Primary sources: Google Search Central: supporting a shared AI training control for websites · IETF AIPREF vocabulary draft (work in progress) · IETF AI Preferences working group documents · Google: robots.txt introduction.