Short answer: a robots.txt for AI crawlers you can defend names each agent, decides by the agent's role (training corpus, search index or user-triggered fetch) rather than by vendor, repeats your private paths in every group it writes, and records a reason beside each rule and a review date at the top. Then you verify it by fetching your pages as each agent, because a CDN or firewall can block an agent the file allows. Below: a complete annotated file, how a crawler picks the rules that apply to it, where to edit it on Shopify, WordPress, Webflow and HubSpot, and a one-minute test.
The file matters less than the discipline around it. An unexplained rule gets copied forward through every migration because nobody dares delete a line they do not understand, and a policy built on last year's agent names looks current while doing something nobody chose.
Decide by role, not by vendor
The major operators run several agents on purpose, so that site owners can make separate decisions about training and about appearing in answers. A rule aimed at a whole vendor collapses those decisions into one. The classic result is a site that meant to decline training use and, in the same edit, deleted itself from an answer engine. The agent most often caught this way is OAI-SearchBot, blocked by sites that only meant to keep GPTBot out of their content.
The same split, with what blocking each group costs you in practice:
| Role | Agents | What blocking costs | Does robots.txt hold? |
|---|---|---|---|
| Training corpus | GPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta), CCBot (Common Crawl) | Nothing in today's answers. It is a decision about training use. | Yes, these are the agents the file is built for. |
| Search and answer index | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, meta-webindexer | Removal from that engine's search results and answers. | Yes. This is where accidental blocks do the most damage. |
| User-triggered fetch | ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher | A person who asks the assistant to read your page gets nothing back. | Not reliably. Each vendor notes these may not follow it when acting for a user. |
| Usage control (not a crawler) | Google-Extended, Applebot-Extended | Google-Extended: removal from grounded Gemini answers, with Google Search unaffected. Applebot-Extended: training only. | It only exists as a robots.txt token; nothing is crawled under that name. |
Each row comes from the operator's own documentation, listed in full at the end of this article. Our AI crawler list keeps the complete table with each vendor's wording, including ad-review agents such as OAI-AdsBot that only matter if you advertise.
Four positions and what each one costs
Most sites end up in one of four positions. Write yours down before touching the file; that sentence becomes the file's header comment.
| Position | Training agents | Search and user agents | Google-Extended | What you give up |
|---|---|---|---|---|
| Open | Allowed | Allowed | Allowed | Any say over training use. |
| Decline training, stay in answers (the file below) | Blocked | Allowed | Allowed, as a recorded decision | Nothing in answers; Gemini may still use content under Google-Extended. |
| Decline training including Gemini | Blocked | Allowed | Blocked | Grounded Gemini answers. Google Search inclusion and ranking are unaffected. |
| Block all AI agents | Blocked | Blocked | Blocked | Presence in ChatGPT search, Perplexity, Claude, Meta AI and grounded Gemini answers, plus Apple's search surfaces. User-triggered fetches may still arrive. |
Two gaps are worth knowing before you choose. xAI does not publish a documented crawler user agent, so Grok cannot be allowed or blocked by name. We could not verify a Copilot-specific crawler in Microsoft's public webmaster documentation either. Do not write rules for guessed names: a rule against a name nothing honours is worse than a gap, because you will believe you have allowed or blocked something.
The file, annotated
This version takes the second position: it declines training use and stays eligible for search answers and user requests. Change the decisions to suit your own position, but keep the structure and the comments.
# AI crawler policy for example.com
# Checked against each vendor's documentation on 2026-09-09.
# Owner: web team. Re-check when a vendor changes its crawler docs.
# --- Training corpora: declined. Does not affect today's answers. ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /
# --- Usage controls (not crawlers) ---
# Applebot-Extended: training only; Apple says search is unaffected.
User-agent: Applebot-Extended
Disallow: /
# Google-Extended also controls grounding in Gemini answers,
# so blocking it would remove us from those answers. Decision: allowed.
User-agent: Google-Extended
Allow: /
# --- Search and answer indexes: allowed. Blocking removes us from answers. ---
# Private paths are repeated here: a named group does not inherit from *.
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Applebot
User-agent: meta-webindexer
Allow: /
Disallow: /admin/
Disallow: /cart/
# --- User-triggered fetchers: allowed. Several may ignore this file anyway. ---
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: meta-externalfetcher
Allow: /
Disallow: /admin/
Disallow: /cart/
# --- Everyone else ---
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
Four details in it are deliberate:
- The private paths are repeated in each named group. A crawler that finds a group naming it follows that group and ignores the
*group entirely. If/admin/and/cart/lived only under*, every search agent you allowed would be allowed into them too. The next section shows this step by step. - Several user agents share one group. Consecutive
User-agentlines form one group with the rules beneath them, so the five search agents get identical treatment from one block. Nothing can drift between them. - Google-Extended is a trade-off, not a free opt-out. Google documents it as controlling both Gemini training and grounding in Gemini Apps and Vertex AI. Blocking it keeps you in Google Search but removes you from grounded Gemini answers, which is why the comment records a decision rather than a default. Gemini and Google-Extended covers that choice in more depth.
- The user-triggered group is allowed, but not relied on. OpenAI, Perplexity, Anthropic and Meta each note that agents fetching on a user's behalf may not follow robots.txt the way crawlers do. If you genuinely need to stop them, that is an edge or firewall decision with a real cost, not a robots.txt line.
If all you want is to opt out of OpenAI's training crawl and nothing else, how to block GPTBot without disappearing walks through that narrower case.
Comments are part of the policy
Vendor documentation changes. New agents appear and existing ones are re-described, so the header carries the date you checked and an owner, and every block carries its reason. Review the file whenever you change a decision, and whenever a vendor updates its crawler page. Two lines of context today save whoever inherits the file from guessing which lines are load-bearing.
How robots.txt for AI crawlers matches a request
Most broken AI policies are not wrong about which agents to block; they are wrong about how a crawler reads the file. A crawler does not apply every line it sees. It picks one set of rules and ignores the rest.
The selection happens in two steps, and both are mechanical:
- Choose the group. The crawler looks for a group whose
User-agentline matches its own name, ignoring case. If one exists, that group is the whole policy for this crawler. If several groups name it, their rules are combined. Only when no group names it does the crawler fall back toUser-agent: *, and if there is no*group either, everything is allowed. - Choose the rule. Inside the chosen group, the rule with the longest matching path wins, regardless of the order the lines are written in.
Disallow: /cart/beatsAllow: /for a cart URL because it is more specific. When an Allow and a Disallow match equally, the Allow wins.
Those two steps explain the failures we see most often in AI-related edits:
- Adding a named group silently opens private paths. Someone adds
User-agent: OAI-SearchBotwithAllow: /to be explicit, and the agent stops reading the*group that held the private paths. - An empty Disallow means the opposite of what it looks like.
Disallow:with nothing after it allows everything;Disallow: /blocks everything. One character separates the two. - Paths are case-sensitive even though agent names are not.
Disallow: /Admin/does not cover/admin/. - Rule order does not matter, so reordering cannot fix a conflict. If an Allow you added is not taking effect, look for a longer Disallow in the same group.
Where to edit it on each platform
On a hand-built site robots.txt is a file in the web root. On the common hosted platforms it is generated, and each one starts from defaults that interact with the matching rules above:
| Platform | Where the rules live | Default you start from | The trap |
|---|---|---|---|
| Shopify | A robots.txt.liquid template in the published theme that iterates the Liquid robots object and adds rules | Disallows /admin, /cart, /checkout, /search, /policies/ and filtered collection URLs | Replacing the whole file loses defaults Shopify updates regularly; only the published theme is served. |
| WordPress | A virtual file from do_robots(), changeable with the robots_txt filter | User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a Sitemap line | On standard nginx or Apache setups a real robots.txt in the web root is served first and overrides the virtual one. |
| Webflow | Site settings, SEO, Indexing, including traffic-control toggles for AI crawler categories | Rules generated from those toggles; read them rather than assume they allow AI agents | The webflow.io staging subdomain can serve a different, blocking file, so audit the custom domain. |
| HubSpot | Settings, Content, Pages, then SEO and Crawlers, per connected domain | System domains containing hs-sites are permanently no-index whatever you write | A blanket disallow also blocks HubSpotContentSearchBot and silently breaks on-site search unless it gets its own allow group. |
On Shopify, keep the default loop and append your AI groups after it. Because an agent you name follows only your group, copy the private paths from the defaults into that group:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
# AI search agents: allowed, with Shopify's private paths repeated
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /policies/
# AI training agents: declined
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
On WordPress, the filter appends to the virtual file, so the same rule applies: a named group needs its own /wp-admin/ line. Check first that no physical robots.txt is sitting in the web root, or the filter will never run.
// In a small plugin or the child theme's functions.php
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) {
return $output; // site is set to discourage indexing; leave it alone
}
$output .= "\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nDisallow: /\n";
$output .= "\nUser-agent: OAI-SearchBot\nUser-agent: PerplexityBot\n"
. "Allow: /\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n";
return $output;
}, 10, 2 );
On Webflow and HubSpot you paste the groups into a settings field (Webflow's details here come from its help centre), then read back what the custom domain serves. HubSpot sites that use its site search should add User-agent: HubSpotContentSearchBot with Allow: / before tightening anything.
Testing robots.txt for AI crawlers by fetching, not reading
Reading robots.txt tells you your intent. Only a request tells you your behaviour. Between an AI agent and your content there are four places it can be stopped, and only the first one is written in the file.
Request one important page once per agent, with an ordinary browser user agent as a control, then fetch the robots.txt your live domain actually serves:
URL=https://example.com/pricing/
for ua in "Mozilla/5.0" GPTBot OAI-SearchBot ChatGPT-User \
PerplexityBot Perplexity-User ClaudeBot Claude-SearchBot Claude-User; do
printf "%-20s " "$ua"
curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" -A "$ua" "$URL"
done
# And the file itself, exactly as your live domain serves it:
curl -s -w "\n-- %{http_code}\n" https://example.com/robots.txt
Read the output against this table:
| What you see | What it means | Where to fix it |
|---|---|---|
| The browser line fails too | The site is down or the URL is wrong. Without this control, an outage looks exactly like a block. | Fix the URL or the outage, then rerun. |
| 403 on an agent you allowed, 200 for the browser | The edge is blocking it, whatever robots.txt says. | CDN, firewall or bot-management rules, not the file. |
| 200 for the agent but a far smaller body than the browser gets | A failure, not a pass. Most AI crawlers do not execute JavaScript, so a script-built page reaches them nearly empty. | Server rendering; see JavaScript rendering and AI crawlers. |
| Matching status and size across all lines | The server side is fine for this page. | Read the served robots.txt against your intended position. |
Two cautions keep this test honest. It shows what your server returns to each agent, not whether a crawler obeys your file, which is why the last command prints the file as actually served. And keep "could not fetch" separate from "not present": a timeout on robots.txt says nothing about whether the file exists.
A curl request from your own machine is also not the real crawler. An edge can recognise known agents by network origin as well as by name, so a mismatch is a lead to follow in your CDN's rules and logs, not a final verdict. If you would rather have these probes run with the browser control and the empty-body check built in, that is what InsightWonder's AI crawlability check does.
Sources
- OpenAI: overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot)
- Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User)
- Anthropic: how site owners can block the crawler (ClaudeBot, Claude-SearchBot, Claude-User)
- Google: common crawlers (Google-Extended, Google-CloudVertexBot)
- Apple: about Applebot (Applebot, Applebot-Extended)
- Meta: web crawlers (meta-externalagent, meta-webindexer, meta-externalfetcher)
- Common Crawl: CCBot
- Shopify: robots.txt.liquid, WordPress: do_robots(), Webflow: set robots.txt rules, HubSpot: customize robots.txt