What Is Google-Extended?
Google-Extended is a dedicated user-agent token introduced by Google in 2023. It allows website owners to control whether Google's AI products, such as Bard and the AI features that power AI Overviews in Search, can crawl and use their site's content for training and improving those AI systems. Crucially, this control operates independently of Googlebot, which handles standard organic search crawling and indexing.
Before Google-Extended existed, website owners had a binary choice: block all Google crawlers and lose search rankings, or allow everything, including AI training. Google-Extended resolves this dilemma by splitting the permissions. You can block Google-Extended in your robots.txt file without affecting how Googlebot indexes your pages for standard search results.
The directive works like any other robots.txt user-agent rule. To block Google-Extended from your entire site, you would add User-agent: Google-Extended followed by Disallow: /. To allow it on specific paths only, you can use more granular rules. This gives South African content publishers, news sites, e-commerce businesses, and agencies meaningful control over how their intellectual property is consumed by AI systems.
It is worth noting that blocking Google-Extended does not guarantee your content will never appear in AI-generated answers. Google may still surface cached versions, publicly available snippets, or information gathered before you added the block. It primarily affects ongoing AI training and feature improvement. Businesses investing in SEO and AI search visibility should weigh this decision carefully against their goals for appearing in AI Overviews.
Google-Extended In Practice
The two scenarios below are illustrative examples, not Juicy Designs client results. The outcomes described indicate the kind of effect that a Google-Extended decision typically produces, so treat them as indicative rather than measured.
Consider a Johannesburg-based legal news publisher that spends significant resources producing original South African case law summaries. Without a Google-Extended rule in place, Google's generative AI systems could use those summaries for model training and grounding, and a publisher in that position might reasonably worry that AI-generated answers built on its work would reduce the click-throughs it depends on.
By adding User-agent: Google-Extended and Disallow: / to its robots.txt, the publisher would block Google's AI training and grounding use while keeping standard Googlebot access intact. Its articles would still be eligible to rank in organic search results, but its content would no longer be fed into Google's AI training pipelines without consent.
On the other side of the equation, imagine a Pretoria-based digital marketing agency that chooses to allow Google-Extended explicitly on its blog and glossary content. The reasoning would be straightforward: it wants its definitions and guides available to Google's generative AI products so that they can be drawn on and cited in AI-generated answers, increasing brand visibility and potentially driving referral traffic. Allowing AI crawlers to access high-quality, well-structured content can improve the chances of being cited as a source in those answers.
What Google-Extended controls
Google-Extended is a control, used through robots.txt, that lets a website manage whether its content can be used to train and ground Google's generative AI models, such as the Gemini apps and Vertex AI, separately from normal Google Search crawling. The important point is that Google-Extended is distinct from Googlebot: allowing or disallowing Google-Extended governs Google's AI training and grounding uses, while Googlebot governs crawling for Google Search, including appearance in Search features. This separation means a site can allow normal search crawling and indexing, and its appearance in Google Search AI features such as AI Overviews, which follow standard Search rules, while independently choosing whether to permit Google's other AI systems to use its content. Google-Extended gives publishers a specific lever over that particular use, reflecting the different considerations around AI training versus search indexing.
Should you allow Google-Extended?
Deciding whether to allow Google-Extended is a genuine trade-off with no universal answer. Allowing it permits Google's generative AI systems to use your content, which can support your presence and representation in those AI products, aligning with a goal of broad AI visibility. Disallowing it opts your content out of that AI training and grounding, which some publishers choose to protect their content or for principled reasons about AI use of their work. Crucially, blocking Google-Extended does not affect your Google Search rankings or your eligibility for Search AI features like AI Overviews, since those are governed by Googlebot and standard Search rules, so the decision is specifically about Google's separate AI systems, not about search visibility. For most businesses seeking maximum AI presence, allowing it is consistent with that aim, but the choice should be made deliberately according to how you weigh AI visibility against control over your content's use.
FAQ
Does blocking Google-Extended affect my Google Search rankings?
No. Google-Extended controls AI training and AI features only. Blocking it does not affect Googlebot crawling or your organic search rankings. Your pages will still be indexed and ranked normally in Google Search results.
Should South African businesses block Google-Extended?
It depends on your content strategy. If you want your brand cited in AI Overviews and Google's generative features, allow Google-Extended. If you want to protect original content from AI training without permission, blocking it is a reasonable choice.
Does blocking Google-Extended affect Google Search rankings?
No. Google-Extended governs Google's generative AI training and grounding, separately from Googlebot, which handles Search crawling. Blocking Google-Extended does not affect your Search rankings or eligibility for Search AI features like AI Overviews, which follow standard Search rules. It only opts your content out of Google's other AI systems.
Should a business block Google-Extended?
It depends on how you weigh AI visibility against control over your content. Allowing it lets Google's generative AI use your content, supporting presence in those products; blocking it opts out to protect content. Since it does not affect Search, the choice is specifically about Google's AI systems, and most businesses seeking AI visibility allow it.