Short answer: block GPTBot and explicitly allow OAI-SearchBot and ChatGPT-User. OpenAI documents them as separate agents with separate jobs: GPTBot crawls content that may be used to train OpenAI's models, OAI-SearchBot builds the index that surfaces websites in ChatGPT's search features, and ChatGPT-User fetches a page because a person asked for it. Blocking GPTBot alone does not remove you from ChatGPT search results. One rule that blocks all three does.
The rest of this article is the file that separates the two decisions, where that file lives on the common site platforms, which agents a file cannot stop, and how to prove your server behaves the way the file says.
Why one broad rule is the expensive way to block GPTBot
Most sites that block AI crawlers are making a decision about training data. That is a legitimate position, and blocking GPTBot is a defensible way to state it. The trouble is the rule people actually write. A wildcard aimed at "OpenAI", a copied list of "AI bots" or a one-click toggle in a CDN usually sweeps up OAI-SearchBot as well. In our own crawler reference it is the agent we flag as the one sites most often block by accident while meaning to opt out of training.
The cost never shows up as an error. No page returns a server error, your analytics look the same, and no one gets an alert. You simply stop being an option when a buyer asks ChatGPT a question your page could have answered. That is why the decision has to be written agent by agent, with the reason next to each rule.
The asymmetry is worth stating plainly. Blocking GPTBot changes nothing in the answers ChatGPT gives today; it is a statement about future training use. Blocking OAI-SearchBot changes what ChatGPT search can show today. A rule that cannot tell the two apart trades a real, current cost for a symbolic one.
The four OpenAI agents and what blocking each one costs
| Agent | What it does | What blocking it costs |
|---|---|---|
GPTBot | Crawls content that may be used to train OpenAI's generative foundation models. | Nothing in today's answers. It is a training-use decision. |
OAI-SearchBot | Builds the index that surfaces websites in ChatGPT's search features. | Removal from ChatGPT search results. |
ChatGPT-User | Fetches a page because a person in ChatGPT or a Custom GPT asked for it. | A user who explicitly asks ChatGPT to read your page gets nothing back. |
OAI-AdsBot | Checks the safety of pages submitted as ChatGPT ads, and visits only those pages. | Only relevant if you advertise in ChatGPT. |
The primary source is OpenAI's bot documentation. We last checked it on 2026-09-09, and vendor pages like this one change, so re-read it before you rely on any single row. The same split exists at other vendors, which is why the file below treats training and retrieval as two groups rather than as one list of "AI bots". Every agent and its source is in our AI crawler list.
The same split at other vendors
OpenAI is not unusual here, which is what makes a per-role file practical. Anthropic separates ClaudeBot, which collects content that may contribute to training, from Claude-SearchBot and Claude-User, which serve search and user requests. Perplexity documents that PerplexityBot indexes pages for its answers and is not used to crawl content for foundation models. Apple keeps Applebot for search and offers Applebot-Extended purely as a training opt-out. Google is the exception that bundles: Google-Extended covers Gemini training and grounding together, so there is no way to decline one and keep the other.
A robots.txt that separates training from retrieval
# Training corpora: declined.
# Reason: our position on content used for model training.
# Owner: web team. Re-check the vendor docs before editing.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Search and user-triggered agents: allowed.
# Blocking these removes us from answers. That is not the intent.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Applebot
Allow: /
Four notes on the shape of this file.
- Every agent name is documented by its operator. Do not add names you have not verified. Inventing a user agent is worse than leaving a gap, because whoever maintains the file will believe a rule is doing something that nothing honours. xAI, for example, publishes no crawler user agent for Grok, so there is nothing to write for it by name.
- CCBot is a judgement call. Common Crawl describes its repository as open to anyone rather than naming AI users, so treat downstream training use as likely, not vendor-confirmed.
- Applebot-Extended is not a crawler. Apple documents it as a training opt-out only; pages that disallow it can still appear in Apple's search results, which is why Applebot itself stays allowed.
- Google-Extended is deliberately missing. It controls Gemini training and grounding together, so blocking it also removes you from grounded Gemini answers. That is a separate decision, covered in Gemini SEO and Google-Extended.
The comments matter more than they look. In a year, whoever opens this file will not remember which line was deliberate, and an unexplained Disallow tends to be copied forward indefinitely. For a longer annotated version covering every vendor, see a robots.txt for AI crawlers you can actually defend.
Where to block GPTBot on each platform
On a hand-built site robots.txt is a file in the web root. On the hosted platforms it is something else, and each has its own way to go wrong:
| Platform | Where the rules live | The way it goes wrong |
|---|---|---|
| Shopify | A robots.txt.liquid template in the published theme that iterates the Liquid robots object and appends your groups | Replacing the file instead of appending loses Shopify's updated defaults; a template in an unpublished theme is never served. |
| WordPress | A virtual file from do_robots(), changed with the robots_txt filter, or a real file in the web root | On standard nginx or Apache setups the real file wins, so a plugin's edits can silently stop applying. |
| Webflow | Site settings, SEO, Indexing, with toggles that generate rules for bot categories including AI crawlers | The toggle's label is not the rule; and the webflow.io staging address can serve a different, blocking file. |
| HubSpot | Settings, Content, Pages, SEO and Crawlers, set per connected domain | A blanket disallow also catches HubSpotContentSearchBot and quietly breaks on-site search. |
Two habits prevent most of these. Always edit on the live, custom domain and then fetch /robots.txt from that domain to see what is really served. And when a platform generates rules for you, read the generated output line by line rather than trusting a toggle called "AI bots", because a single category can contain training, search and user agents together. Shopify's own instructions are in its robots.txt.liquid documentation.
The agents robots.txt cannot stop
OpenAI, Anthropic and Perplexity each note that their user-triggered agents may not follow robots.txt the way crawlers do, because a person made the request. That covers ChatGPT-User, Claude-User and Perplexity-User; Meta says the same of meta-externalfetcher. Treat robots.txt as a statement of intent for crawlers, not as access control.
If you genuinely need to stop one of these agents, that is an edge-rule decision at your CDN or firewall, and it fails a request a real person made. If you do write edge rules, match on more than the user-agent header, which any client can send. OpenAI publishes address lists for GPTBot, OAI-SearchBot and ChatGPT-User, which let you write rules against the real agents and check your access logs for them.
Verify the GPTBot block: fetch as each agent, not by reading the file
Reading robots.txt tells you what you intended. Fetching as each agent tells you what your server does. A CDN rule or bot-management product that returns 403 to a named agent is a block in practice, and the file will keep saying "Allow".
# A browser-like control first, so "site down" is not mistaken for "blocked"
curl -s -o /dev/null -w "browser %{http_code}\n" -A "Mozilla" https://example.com/pricing/
# Then each OpenAI agent: status code and how much text came back
for ua in GPTBot OAI-SearchBot ChatGPT-User; do
printf "%-14s" "$ua"
curl -s -A "$ua" -w " %{http_code}\n" https://example.com/pricing/ | wc -c
done
Read the output with three rules, in this order:
- Run the browser control alongside the agents. Without it, a site that is simply down looks identical to a site that blocks AI agents.
- Treat a 403 for OAI-SearchBot or ChatGPT-User as a real problem. Something in front of your site is overriding the file.
- Treat a 200 with almost no bytes as a failure, not a pass. Most AI crawlers do not execute JavaScript, so a page that only renders client-side is an empty shell to them. Our guide to JavaScript rendering and AI crawlers covers the fixes.
This test proves what your server returns. It cannot prove that GPTBot honours your Disallow; only your access logs, checked against OpenAI's published addresses, can show that over time. A request that says GPTBot but comes from an address outside gptbot.json is not OpenAI's and tells you nothing about its compliance; filter those out before drawing any conclusion. If you would rather not script the probes, our AI crawlability check runs the same agent fetches with a browser control group.