Definition
Google-Extended is not a crawler. It is a usage control: it determines whether content Google has already crawled may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google documents that it does not affect inclusion or ranking in Google Search.
Why it matters
It is the cleanest separation any vendor currently offers between search presence and AI use, which makes it the one place where the trade-off can be reasoned about rather than guessed at. It also means a Disallow here has a specific, bounded cost — grounded Gemini answers — and not the vague cost people assume.
How to measure it
Because it controls usage rather than fetching, you cannot verify it by watching server logs. Treat it as a stated policy decision, record it, and measure the effect on the answer side: mention and citation rates in Gemini over a fixed question set, compared against engines where no such control was applied.