Google backs shared AI training control in robots.txt

On 30 October 2026, Google Search Central announced plans to support a shared AI training control in robots.txt, based on work by the AIPREF working group at the Internet Engineering Task Force (IETF). Websites will be able to state broad AI training preferences with a single policy token, alongside existing controls like Google-Extended. Google aims to support this with other AI providers in 2027.

Key takeaways

• Announced by Google Search Central on 30 October 2026.
• A shared policy token in robots.txt will express AI training preferences across participating providers.
• If you opt out with the token and don’t specify Google-Extended, Google plans to treat it as disallowed for Google-Extended.
• Company-specific controls (like Google-Extended) will continue to work alongside the shared signal.
• Google aims to support the new control with other AI providers in 2027; it aligns with IETF AIPREF and the EU GPAI Code of Practice.

Google’s announcement and why it matters for B2B

On 30 October 2026, Google Search Central announced plans to support a shared AI training control in robots.txt so websites can express training preferences in one place. The proposal draws on work from the AIPREF working group at the Internet Engineering Task Force (IETF) and reflects input from publishers, civil society, regulators and others seeking industry-wide controls.

Google notes the approach is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act, which recognises robots.txt as a primary way to state machine-readable preferences. Google aims to support the new control with other AI providers in 2027, and will share progress updates.

For B2B organisations, this offers clearer governance over content use in AI training without juggling multiple vendor-specific instructions. It also preserves room for provider-level choices. ARTIC follows these developments closely to help teams plan policy, compliance and visibility strategies across web and AI experiences.

How the shared policy token is expected to work

The new mechanism introduces a shared ‘policy token’ in robots.txt so site owners can set broad AI training preferences once across participating AI providers.

Key points from Google’s announcement:

• Unified opt‑out: if a website uses the shared token to opt out of training and does not explicitly specify Google‑Extended, Google plans to interpret this as disallowing Google‑Extended.
• Cross‑industry handling: other AI providers can handle it similarly, making it easier to opt out across all or to pick and choose.
• Works with existing controls: company‑specific controls will continue alongside the shared signal, allowing a broad choice plus fine‑tuning for individual AI developers (e.g., Google‑Extended).
• AI experiences: websites that specifically allow Google‑Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

This provides an early, shared preference signal based on the IETF’s AI training definition while multistakeholder discussions continue on the broader AIPREF vocabulary.

What it means for B2B sites: SEO, AI exposure and governance

The announcement gives B2B teams a practical path to centralise AI training preferences in robots.txt while retaining per‑provider nuance. You could set a broad default position and adjust exceptions for specific AI developers where participation offers value, such as potential visibility via Gemini Apps when Google‑Extended is allowed.

For SEO, this is about AI training preferences, not replacing standard crawling and indexing rules. Treat it as an additional policy layer and keep your crawl directives clear and intact. It’s also a moment to align legal, compliance and brand on acceptable use of content in model training.

If you’re exploring generative search and assistants, see our overview of Generative Engine Optimisation (GEO) basics to understand how content can appear in AI experiences. ARTIC can help teams balance AI visibility opportunities with content protection and regulatory context.

What to do now: prepare your robots.txt and policies

Move ahead with a structured plan:

• Decide your default stance on AI training (allow, selective allow, or opt out) and document exceptions by section or path.
• Inventory sensitive content (e.g., product documentation, gated resources) versus content intended for broad discovery.
• Review current robots.txt and existing controls (including any Google‑Extended settings) and clarify internal priorities.
• Coordinate with Legal/Compliance to align on risk, contracts and licensing posture.
• Plan implementation and change control so the shared token can be added cleanly once supported; version and test before publishing.
• Monitor vendor updates from the IETF AIPREF working group and Google as 2027 support approaches.

For execution discipline, revisit Technical SEO for safe robots.txt changes, consider an Audit of your SEO and AI exposure, and streamline collaboration with secure access to Search Console and Ads tools.

Implement carefully without harming crawling and indexing

Although the shared control is a proposal for future support, you can set yourself up for a smooth adoption:

• Keep layers separate: maintain your crawling/indexing rules as they are, and manage AI training preferences as a distinct policy layer.
• Be explicit where needed: if you rely on Google‑Extended for certain surfaces, state it explicitly. Remember Google plans to treat a shared‑token opt‑out with no explicit Google‑Extended as a disallow for Google‑Extended.
• Define exceptions by structure: use clear patterns for directories or sections likely to diverge from your default stance.
• Document ownership and versioning: keep comments in robots.txt, track changes, and assign responsibility for upkeep.
• Watch industry adoption: as other AI providers handle the shared token, adjust your mix of the shared preference and provider‑specific controls.

This approach helps you integrate the upcoming shared policy token while protecting crawl health and balancing AI visibility with content governance.

Frequently asked questions

What exactly did Google announce, and when?

On 30 October 2026, Google Search Central announced plans to support a shared AI training control in robots.txt so websites can express training preferences in one place, based on work by the AIPREF working group at the IETF.

How does this interact with Google‑Extended and Gemini Apps?

If you use the shared token to opt out and don’t explicitly specify Google‑Extended, Google plans to interpret that as disallowed for Google‑Extended. Sites that specifically allow Google‑Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Does the shared control replace existing robots.txt rules?

No. Google says company‑specific controls will continue to function alongside the shared signal. You can state a broad preference and still tailor instructions for individual AI developers like Google.

When will this be supported?

Google aims to support the new control with other AI providers in 2027 and will provide updates on progress. You can prepare now by defining policy and planning robots.txt changes.

How does this affect your website?

If you want to know whether your website is ready, ARTIC runs SEO audits, technical SEO and generative engine optimisation (GEO) for B2B companies, with monthly reports that track leads, not just visits.

Sources

• Supporting a shared AI training control for websites | Google Search Central Blog | Google for Developers

Informative article with the sources linked, up to date on the publication date.