Letting AI crawlers in: robots.txt, GPTBot and the checks that actually prove it

Short answer: an answer engine cannot cite a page it cannot fetch, so AI crawler access is the one part of GEO that is a prerequisite rather than an improvement. Make two separate decisions, one about training and one about answers: you can block the training crawlers and still allow the search and user-triggered agents that put you in answers. Then prove access by requesting real pages with each agent's user agent, because a permissive robots.txt does not show that anything gets through.

This guide covers which agents do which job, a robots.txt that writes both decisions down, where that file lives on Shopify, WordPress, Webflow and HubSpot, why reading the file proves nothing, the probe that does prove it, and how to stop access from quietly breaking again a month later.

AI crawler access is two decisions, not one

The agents AI companies run do different jobs, and their vendors document them separately. Some gather content that may be used to train models. Some build the index an assistant searches when it answers. Some fetch a page only because a person just asked about it. A few are not crawlers at all, but controls over how content fetched by another crawler may be used.

Most sites make one decision and apply it to every row, usually by accident, because the agents sit next to each other in the same file. OAI-SearchBot is the agent sites most often block while meaning to opt out of training. Crossing the two decisions gives four positions, and only one of them is usually what a business that wants to be recommended actually means:

The top-right square is worth a second look, because it is where an unreviewed file most often lands. A site blocks OAI-SearchBot because the name sounds like a scraper, while GPTBot, the actual training crawler, is never mentioned and stays allowed by the * group. The result is the opposite of what anyone intended: content still available for training, and the site gone from ChatGPT's search answers. Reading the file one group at a time, and writing next to each line which job that agent does, is what catches it.

Separate the rows deliberately, and write down which square you chose before you touch the file. The full list, with each vendor's purpose statement and blocking cost, is in our AI crawler list.

A robots.txt that encodes both decisions

This file opts out of training for OpenAI and Anthropic, which run separate training crawlers, and stays eligible for their answers. Adapt it to your own position; the point is that each agent gets an explicit decision.

# Training: opted out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Optional: opts out of Gemini training, but also removes you from grounded Gemini answers
# User-agent: Google-Extended
# Disallow: /

# Search and answers: allowed
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/

# User-triggered fetches: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /
Disallow: /admin/

User-agent: *
Disallow: /admin/

Notice that /admin/ is repeated in every named group. A crawler that finds a group naming it follows that group and ignores the * group entirely, so a private path listed only under * is open to every agent you name. This is the most common way an "allow AI search" edit accidentally opens a private section.

Google-Extended is left commented out on purpose. Uncommenting it opts your content out of Gemini training and does not affect Google Search, but it also removes you from grounded Gemini answers. Enable it only if the training question matters more to you than appearing in Gemini's answers. How to block GPTBot without disappearing works through these trade-offs.

Two engines cannot be handled by name. xAI does not publish a documented crawler user agent, so Grok cannot be allowed or blocked by name, and we could not verify a Copilot-specific crawler in Microsoft's public webmaster documentation. For both, what you control is whether your pages are readable to general crawling at all.

Where the file lives on each platform

On a hand-built site robots.txt is a file in the web root. On hosted platforms it is generated or set in an admin screen, and each one starts from a different default:

Whichever platform you use, audit the file your live custom domain actually serves, not the one in the editor or on a staging host.

Why reading robots.txt proves nothing about AI crawler access

robots.txt is a declaration that well-behaved crawlers honour. It is neither enforcement nor the only thing between an agent and your page. Three things make it an unreliable witness:

1. Blocks below the file. A permissive robots.txt in front of a CDN rule, a bot-management product or a firewall that returns 403 to those agents is a block in practice, and the file will keep saying otherwise.
2. Empty responses. Most AI crawlers do not execute JavaScript. A page that returns 200 with its content rendered client-side gives the agent nothing, so a 200 with an empty body is a failure, not a pass. JavaScript rendering and AI crawlers covers the fixes.
3. User-triggered agents. OpenAI, Anthropic, Perplexity and Meta each note that their user-triggered agents may not follow robots.txt the way crawlers do, because a person made the request. robots.txt states your intent; it does not control access.

The check that actually proves AI crawler access

Request a real content page with each user agent you care about, record the status code and the response size, and search the body for a sentence you know is on the page:

URL="https://example.com/guides/returns-policy/"
PHRASE="refund"
for UA in "OAI-SearchBot" "ChatGPT-User" "PerplexityBot" \
          "Claude-SearchBot" "Claude-User" "Mozilla"; do
  BODY=$(curl -s -A "$UA" -w "\n%{http_code}" "$URL")
  CODE=$(echo "$BODY" | tail -n1)
  HIT=$(echo "$BODY" | grep -ci "$PHRASE")
  echo "$UA  status=$CODE  phrase_hits=$HIT  bytes=${#BODY}"
done

Read each line of output with the same three questions, in this order:

The last line in the loop, the plain browser-style user agent, is the control group, which is why it is the first question. Without it, a site that is simply down looks identical to a site that blocks AI agents. Treat "could not fetch" as unknown, never as "not present". Run the check against a content page, not only the homepage, because the pages that answer buyer questions are often the ones behind a framework.

A spoofed user agent tests your rules, not the vendor's real traffic. An edge can also recognise known agents by where they connect from, so a mismatch between your probe and real crawler behaviour is a lead to follow in your CDN's logs, not a final verdict. To confirm that a request claiming to be an OpenAI agent really came from OpenAI, compare its source address with the published IP lists, such as the OAI-SearchBot list.

Keep AI crawler access from silently breaking

Fix a block at the layer that causes it, which is often not robots.txt. Then recheck on a schedule, because each layer between an agent and your content tends to be owned by a different team and to change for reasons that have nothing to do with AI:

That spread of ownership is why access breaks quietly. A security team tightens a bot rule after an unrelated incident; a front-end rebuild moves product copy into a client-side component; a vendor documents a new agent. Nobody involved is thinking about answer engines, and nothing in your analytics will flag the change. Write the date next to every result: crawler access is not a state you reach once. The agent list changes too; the table this guide draws on was checked against vendor documentation on 2026-09-09. If you would rather run the probe with a browser control group automatically, that is what our AI crawlability check does.

Sources

• OpenAI: overview of OpenAI crawlers, with IP lists such as searchbot.json
• Perplexity: Perplexity crawlers
• Anthropic: how site owners can block the crawler
• Google: common crawlers
• Meta: web crawlers
• Shopify: robots.txt.liquid, WordPress: do_robots(), Webflow: set robots.txt rules, HubSpot: customize robots.txt