In one sentence: an answer engine cannot cite a page it cannot fetch, so crawler access is the one part of GEO that is a prerequisite rather than an improvement.
Two decisions, not one
The agents that AI companies run do different jobs. Some gather content for training corpora. Others fetch a page because a user just asked a question that needs it. Blocking the first is a defensible business decision about your content being used to train a model. Blocking the second removes you from answers you would otherwise appear in, this week, for buyers who are actively asking.
Most sites make one decision and apply it to both, usually by accident, because the two kinds of agent are listed next to each other in the same file. Separate them deliberately.
Training access and retrieval access are different questions. Answer them one at a time.
The agents worth naming
OpenAI runs GPTBot for content gathering, OAI-SearchBot for search indexing and ChatGPT-User for fetches triggered by a user's request. Perplexity runs PerplexityBot, Anthropic runs ClaudeBot, Google's Google-Extended governs use in its AI products separately from Googlebot, and Common Crawl's CCBot feeds many downstream datasets. Chinese-market assistants run their own, including Bytespider.
Name the ones you have a view on and let the rest fall to your default rule. An agent you have never heard of is a review item, not automatically a threat.
Why reading robots.txt proves nothing
robots.txt is a declaration that well-behaved crawlers honour. It is not enforcement, and it is not the only thing standing between an agent and your page. A permissive robots.txt in front of a CDN rule, a bot-management product or a firewall that returns 403 to those user agents is a block in practice, and the file will keep saying otherwise.
Two more traps are common. The first is precedence: the most specific matching group wins, so a later wildcard group does not apply to an agent that has its own group, and a rule you thought was global may not be. The second is that a fetch can succeed and still be useless, because the response is a shell whose content only appears after JavaScript runs.
The check that actually proves it
Request a real page with each user agent you care about and record the status code and the size of the response body. Then compare that body against what a browser renders. Three outcomes matter: a 403 or 429 means you are blocked somewhere below robots.txt; a 200 with a near-empty body means the content is client-rendered and the agent gets nothing; a 200 with the text present means you are genuinely readable.
Run it against a content page, not the homepage. Homepages are often static while the pages that answer buyer questions are the ones behind a framework.
What to do with the result
Fix blocks at the layer that causes them, which is usually not robots.txt. Then re-check, because bot-management rules are frequently updated by a different team than the one that owns the site. Record the date of the check next to the result: crawler access is not a state you achieve once, it is a state that quietly changes.