Google-Extended
A robots.txt control that lets a site block or allow Google's AI training crawler, separately from regular Googlebot.
What is Google-Extended?
Google-Extended is a setting a website can add to its robots.txt file to specifically control whether Google can use its content to train and improve AI systems like Gemini, separately from whether the site shows up in regular Google Search results. A site could, for example, stay fully visible in Google Search while opting its content out of being used for Gemini's AI training. This separation matters because many sites want to keep their search visibility but have concerns about AI training.
Why it matters
Unlike GPTBot or Googlebot, Google-Extended is not itself a crawler with its own HTTP user-agent making requests; it is a robots.txt "product token" that Google applies retroactively to content already fetched by its existing Google crawlers, determining whether that content is eligible for uses including future Gemini model training, grounding responses in the Gemini app, and grounding in Google Search on Vertex AI. A site adds a "User-agent: Google-Extended" block with "Disallow: /" to opt out entirely, or specific paths to opt out selectively, and Google explicitly documents that this control has no effect on a page's inclusion or ranking in standard Google Search. Google introduced the token in September 2023 as generative AI training data debates intensified, giving publishers a more granular opt-out than blocking Googlebot entirely.