What Is Google-Extended?

Google-Extended is a robots.txt token, not a separate crawler, that publishers use to control whether content Google crawls can be used to train future Gemini models and for grounding. Grounding means supplying Search index content to the model at the moment it answers. For a health practice, it is an opt-out control for Gemini training and grounding, and Google states that it does not impact a site's inclusion in Google Search.

How Google-Extended works

Most crawler controls work by naming a bot. Google-Extended is different. Google calls it a standalone product token, and says it has no HTTP request user agent string of its own. Google's existing crawlers do the fetching. The token is read as a control signal in your robots.txt and nothing else.

Two ideas need defining here, because the page uses them often.

• Gemini is Google's family of AI models. Training is how a model learns its background knowledge, and that knowledge only changes when a new model version ships (see training data). According to Google, the models trained under this control power Gemini Apps and the Vertex AI API for Gemini.
• Grounding means supplying content from Search's index to the model when it answers a prompt, so the answer rests on real pages. For more, see grounding.

If you add a Google-Extended rule, you are asking Google not to use your content for those two purposes. You are not asking Google to stop indexing your site. Googlebot, the crawler for Search, has its own token, and rules for Googlebot apply to Google Search and its features. Google's documentation says those rules apply to Search including Discover and other Search features.

Where do AI Overviews and AI Mode fit? Google's AI features page is direct about it. There are no additional requirements to appear in AI Overviews or AI Mode, and no special markup or AI text files are needed. A page must be indexed and eligible to show with a snippet. To limit what appears in Search, Google points to nosnippet, data-nosnippet, max-snippet and noindex. For limiting AI training and grounding in Google's other systems, it points to Google-Extended. So the two controls are on different layers. AI Overviews follow the Search rules, and Gemini training and grounding follow the token.

Google's page also names its other crawlers, and the one to remember is Googlebot, which has a Smartphone and a Desktop version. Rules for Googlebot apply to Google Search and to products such as Discover, Google Images and Google News. That is why a careless rule aimed at AI can do real damage if it catches Googlebot.

One more fact worth keeping. Google's robots.txt documentation says the instructions in robots.txt files cannot enforce crawler behavior. Compliant crawlers like Googlebot honor them, but the file is a request and not a lock. It also says robots.txt is not a mechanism for keeping a page out of Google. To do that you use noindex or password protection.

Example: Google-Extended at a health practice

Imagine a med spa with detailed treatment guides written by its medical director. The guides explain what each procedure involves, who is a candidate and what recovery looks like. This example is illustrative, not a transcript of a real site.

The owner has two wishes at once. She wants the guides to keep ranking on Google, because that is where patients search. She also does not want the guides used to train Google's future models.

Her developer reads Google's common crawlers page and sees that Google-Extended is the token that expresses the second wish, and that Google says it does not affect inclusion in Google Search. He adds this rule.

User-agent: Google-Extended
Disallow: /

He leaves every Googlebot rule exactly as it was. The guides stay indexed. The owner opens the live robots.txt, sees the rule, and writes the decision and the date in the practice's site notes.

A second owner, a multi-location dental group, has the opposite view. Its pages are public marketing, and it wants every possible mention. It sets no Google-Extended rule and writes that down as a decision too.

Now the wrong version. A third owner reads that AI Overviews are taking clicks, hears that Google-Extended is the "AI switch," and asks the developer to block it. The owner believes the practice is now protected from AI Overviews. It is not, because Google lists AI Overviews and AI Mode among Search features that are governed by Googlebot rules and snippet controls. The owner has changed the Gemini training and grounding setting and nothing else.

The worst version is a developer who, trying to help, blocks Googlebot as well. That one can remove pages from search results. It is a different token and a different job, so check for it.

In all three cases the lesson is the same. Know which question each token answers, and make the choice on purpose.

How to check yours in 60 seconds

1. Open yourpractice.com/robots.txt in a browser.
2. Search the page for Google-Extended.
3. If "User-agent: Google-Extended" has "Disallow: /" below it, you have opted out of Gemini training and grounding.
4. If the name is missing, you have set no rule.
5. Search the page for Googlebot. If it has "Disallow: /" under it, stop and call your developer, because that can pull your site out of Google Search.
6. Write the date and what you found in your site notes.

What to do about it

1. Decide in plain words whether you are comfortable with Google using your public content for Gemini training and grounding.
2. If you want to opt out, add the Google-Extended rule shown above.
3. Leave your Googlebot rules alone, so Search keeps indexing your pages.
4. If you want to limit what Search shows from specific pages, use snippet controls such as nosnippet or max-snippet on those pages instead. See AI bot blocking for how the pieces fit together.
5. Ask your developer to confirm the live file after the change.
6. Write the decision and the date in your site notes so the next rebuild keeps it.
7. Check Search Console after any robots.txt edit to confirm your key pages are still indexed.
8. If you opt out, tell whoever manages your site, so a rebuild does not drop the rule.
9. Recheck the file every quarter, since plugins and host changes can rewrite it.
10. Name one person who approves robots.txt changes. A marketer who wants visibility and a developer or security plugin that blocks bots can pull in opposite directions, and nobody notices until pages drop.