AI crawler directory / Google-Extended
Google-Extended
Documented Provider-maintained crawler reference.
Content ID: OFB-C009
Google-Extended is a standalone robots.txt product token, not a separate page crawler. Google says it controls whether content already crawled by Google may be used for future Gemini model training and certain grounding uses, and that it does not affect inclusion or ranking in Google Search.
Reference facts
- Provider
- robots.txt token
Google-Extended- Purpose
- Data-use control
- Crawls pages
- No
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Not direct
- Status
- active
- Last verified
- Revision
- 1.0
What it controls
- Whether Google-crawled content may be used for future generations of Gemini models covered by Google's documentation.
- Certain grounding uses in Gemini Apps and Grounding with Google Search on Vertex AI.
What it does not control
- Googlebot crawling or a site's inclusion in Google Search.
- Eligibility for AI Overviews or AI Mode; Google documents Googlebot and standard Search controls for those features.
- A separate HTTP request identity, because Google-Extended has no separate user-agent string.
Recommended policy notes
- Treat Google-Extended as a product/data-use control rather than as a separate crawler with its own HTTP user-agent string.
- Use Googlebot and supported preview controls for Google Search and its AI features.
Example policies
Keep Google Search access while restricting covered Gemini uses
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /Google documents that Google-Extended does not affect Search inclusion or ranking.
Allow the Google-Extended product token
User-agent: Google-Extended
Allow: /Common misconceptions
- Google-Extended crawls pages with a separate HTTP user-agent.
- Blocking Google-Extended removes a site from Google Search, AI Overviews, or AI Mode.
How OpenForBots evaluates it
- Parses the Google-Extended product-token group in robots.txt and reports the matching rule.
- Labels the result as a data-use preference rather than a crawler-access test.
- Does not deduct search-visibility points solely because Google-Extended is blocked.
Limitations
- OpenForBots can read the published preference but cannot determine how specific content has been used by Google systems.
- Because Google-Extended is a control token rather than a separate crawler identity, it should not be verified by looking for a Google-Extended HTTP user-agent.
- Google product documentation can change; this reference requires periodic re-verification.
Related guides and concepts
- AI Search Crawlers vs Training Crawlers
- Allow AI Search While Blocking Model Training
- What Makes a Website Understandable to AI?