Short answer: a robots.txt for AI crawlers you can defend names each agent, decides by the agent's role (training corpus, search index or user-triggered fetch) rather than by vendor, repeats your private paths in every group it writes, and records a reason beside each rule and a review date at the top. Then you verify it by fetching your pages as each agent, because a CDN or firewall can block an agent the file allows. Below: a complete annotated file, how a crawler picks the rules that apply to it, where to edit it on Shopify, WordPress, Webflow and HubSpot, and a one-minute test.

The file matters less than the discipline around it. An unexplained rule gets copied forward through every migration because nobody dares delete a line they do not understand, and a policy built on last year's agent names looks current while doing something nobody chose.

Decide by role, not by vendor

The major operators run several agents on purpose, so that site owners can make separate decisions about training and about appearing in answers. A rule aimed at a whole vendor collapses those decisions into one. The classic result is a site that meant to decline training use and, in the same edit, deleted itself from an answer engine. The agent most often caught this way is OAI-SearchBot, blocked by sites that only meant to keep GPTBot out of their content.

TrainingBuilds model corporaAGENTSGPTBotClaudeBotmeta-externalagentCCBotIF YOU BLOCK ITNothing changes intoday’s answersROBOTS.TXTBlockable by nameSearch indexPuts you in answersAGENTSOAI-SearchBotPerplexityBotClaude-SearchBotApplebotmeta-webindexerIF YOU BLOCK ITRemoved from thatengine’s answersROBOTS.TXTBlockable by nameUser fetchA person asked for itAGENTSChatGPT-UserPerplexity-UserClaude-Usermeta-externalfetcherIF YOU BLOCK ITThe user’s requestcomes back emptyROBOTS.TXTMay ignore it ona user’s behalfUsage controlA token, not a crawlerAGENTSGoogle-ExtendedApplebot-ExtendedIF YOU BLOCK ITGoogle-Extended:no Gemini groundingROBOTS.TXTOnly exists as arobots.txt token
Figure 1. The four kinds of AI agent that have official documentation, what blocking each one costs, and where robots.txt stops being reliable. Source: crawler documentation from OpenAI, Perplexity, Anthropic, Google, Apple, Meta and Common Crawl, checked 2026-09-09.

The same split, with what blocking each group costs you in practice:

RoleAgentsWhat blocking costsDoes robots.txt hold?
Training corpusGPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta), CCBot (Common Crawl)Nothing in today's answers. It is a decision about training use.Yes, these are the agents the file is built for.
Search and answer indexOAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, meta-webindexerRemoval from that engine's search results and answers.Yes. This is where accidental blocks do the most damage.
User-triggered fetchChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcherA person who asks the assistant to read your page gets nothing back.Not reliably. Each vendor notes these may not follow it when acting for a user.
Usage control (not a crawler)Google-Extended, Applebot-ExtendedGoogle-Extended: removal from grounded Gemini answers, with Google Search unaffected. Applebot-Extended: training only.It only exists as a robots.txt token; nothing is crawled under that name.

Each row comes from the operator's own documentation, listed in full at the end of this article. Our AI crawler list keeps the complete table with each vendor's wording, including ad-review agents such as OAI-AdsBot that only matter if you advertise.

Four positions and what each one costs

Most sites end up in one of four positions. Write yours down before touching the file; that sentence becomes the file's header comment.

PositionTraining agentsSearch and user agentsGoogle-ExtendedWhat you give up
OpenAllowedAllowedAllowedAny say over training use.
Decline training, stay in answers (the file below)BlockedAllowedAllowed, as a recorded decisionNothing in answers; Gemini may still use content under Google-Extended.
Decline training including GeminiBlockedAllowedBlockedGrounded Gemini answers. Google Search inclusion and ranking are unaffected.
Block all AI agentsBlockedBlockedBlockedPresence in ChatGPT search, Perplexity, Claude, Meta AI and grounded Gemini answers, plus Apple's search surfaces. User-triggered fetches may still arrive.

Two gaps are worth knowing before you choose. xAI does not publish a documented crawler user agent, so Grok cannot be allowed or blocked by name. We could not verify a Copilot-specific crawler in Microsoft's public webmaster documentation either. Do not write rules for guessed names: a rule against a name nothing honours is worse than a gap, because you will believe you have allowed or blocked something.

The file, annotated

This version takes the second position: it declines training use and stays eligible for search answers and user requests. Change the decisions to suit your own position, but keep the structure and the comments.

# AI crawler policy for example.com
# Checked against each vendor's documentation on 2026-09-09.
# Owner: web team. Re-check when a vendor changes its crawler docs.

# --- Training corpora: declined. Does not affect today's answers. ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# --- Usage controls (not crawlers) ---
# Applebot-Extended: training only; Apple says search is unaffected.
User-agent: Applebot-Extended
Disallow: /

# Google-Extended also controls grounding in Gemini answers,
# so blocking it would remove us from those answers. Decision: allowed.
User-agent: Google-Extended
Allow: /

# --- Search and answer indexes: allowed. Blocking removes us from answers. ---
# Private paths are repeated here: a named group does not inherit from *.
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Applebot
User-agent: meta-webindexer
Allow: /
Disallow: /admin/
Disallow: /cart/

# --- User-triggered fetchers: allowed. Several may ignore this file anyway. ---
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: meta-externalfetcher
Allow: /
Disallow: /admin/
Disallow: /cart/

# --- Everyone else ---
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Four details in it are deliberate:

  • The private paths are repeated in each named group. A crawler that finds a group naming it follows that group and ignores the * group entirely. If /admin/ and /cart/ lived only under *, every search agent you allowed would be allowed into them too. The next section shows this step by step.
  • Several user agents share one group. Consecutive User-agent lines form one group with the rules beneath them, so the five search agents get identical treatment from one block. Nothing can drift between them.
  • Google-Extended is a trade-off, not a free opt-out. Google documents it as controlling both Gemini training and grounding in Gemini Apps and Vertex AI. Blocking it keeps you in Google Search but removes you from grounded Gemini answers, which is why the comment records a decision rather than a default. Gemini and Google-Extended covers that choice in more depth.
  • The user-triggered group is allowed, but not relied on. OpenAI, Perplexity, Anthropic and Meta each note that agents fetching on a user's behalf may not follow robots.txt the way crawlers do. If you genuinely need to stop them, that is an edge or firewall decision with a real cost, not a robots.txt line.

If all you want is to opt out of OpenAI's training crawl and nothing else, how to block GPTBot without disappearing walks through that narrower case.

Comments are part of the policy

Vendor documentation changes. New agents appear and existing ones are re-described, so the header carries the date you checked and an owner, and every block carries its reason. Review the file whenever you change a decision, and whenever a vendor updates its crawler page. Two lines of context today save whoever inherits the file from guessing which lines are load-bearing.

How robots.txt for AI crawlers matches a request

Most broken AI policies are not wrong about which agents to block; they are wrong about how a crawler reads the file. A crawler does not apply every line it sees. It picks one set of rules and ignores the rest.

OAI-SearchBot requests /cart/Does any group name OAI-SearchBot?User-agent lines are matched case-insensitivelyNoUse the * groupno * group: all allowedYesFollow that group onlyThe * group is ignored completelyMost specific path rule winsDisallow: /cart/ is longer than Allow: /Result: /cart/ is not fetchedThe common trapPaths you only put inthe * group do notcarry over to namedgroups. Repeat themin every group youwrite for an agent.
Figure 2. How a crawler applies robots.txt to a single request: it first picks one group, then the most specific path rule inside that group. Source: the matching rules of the robots exclusion standard, applied to the example file in this article.

The selection happens in two steps, and both are mechanical:

  1. Choose the group. The crawler looks for a group whose User-agent line matches its own name, ignoring case. If one exists, that group is the whole policy for this crawler. If several groups name it, their rules are combined. Only when no group names it does the crawler fall back to User-agent: *, and if there is no * group either, everything is allowed.
  2. Choose the rule. Inside the chosen group, the rule with the longest matching path wins, regardless of the order the lines are written in. Disallow: /cart/ beats Allow: / for a cart URL because it is more specific. When an Allow and a Disallow match equally, the Allow wins.

Those two steps explain the failures we see most often in AI-related edits:

  • Adding a named group silently opens private paths. Someone adds User-agent: OAI-SearchBot with Allow: / to be explicit, and the agent stops reading the * group that held the private paths.
  • An empty Disallow means the opposite of what it looks like. Disallow: with nothing after it allows everything; Disallow: / blocks everything. One character separates the two.
  • Paths are case-sensitive even though agent names are not. Disallow: /Admin/ does not cover /admin/.
  • Rule order does not matter, so reordering cannot fix a conflict. If an Allow you added is not taking effect, look for a longer Disallow in the same group.

Where to edit it on each platform

On a hand-built site robots.txt is a file in the web root. On the common hosted platforms it is generated, and each one starts from defaults that interact with the matching rules above:

PlatformWhere the rules liveDefault you start fromThe trap
ShopifyA robots.txt.liquid template in the published theme that iterates the Liquid robots object and adds rulesDisallows /admin, /cart, /checkout, /search, /policies/ and filtered collection URLsReplacing the whole file loses defaults Shopify updates regularly; only the published theme is served.
WordPressA virtual file from do_robots(), changeable with the robots_txt filterUser-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a Sitemap lineOn standard nginx or Apache setups a real robots.txt in the web root is served first and overrides the virtual one.
WebflowSite settings, SEO, Indexing, including traffic-control toggles for AI crawler categoriesRules generated from those toggles; read them rather than assume they allow AI agentsThe webflow.io staging subdomain can serve a different, blocking file, so audit the custom domain.
HubSpotSettings, Content, Pages, then SEO and Crawlers, per connected domainSystem domains containing hs-sites are permanently no-index whatever you writeA blanket disallow also blocks HubSpotContentSearchBot and silently breaks on-site search unless it gets its own allow group.

On Shopify, keep the default loop and append your AI groups after it. Because an agent you name follows only your group, copy the private paths from the defaults into that group:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

# AI search agents: allowed, with Shopify's private paths repeated
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /policies/

# AI training agents: declined
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

On WordPress, the filter appends to the virtual file, so the same rule applies: a named group needs its own /wp-admin/ line. Check first that no physical robots.txt is sitting in the web root, or the filter will never run.

// In a small plugin or the child theme's functions.php
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return $output; // site is set to discourage indexing; leave it alone
    }
    $output .= "\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nDisallow: /\n";
    $output .= "\nUser-agent: OAI-SearchBot\nUser-agent: PerplexityBot\n"
             . "Allow: /\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n";
    return $output;
}, 10, 2 );

On Webflow and HubSpot you paste the groups into a settings field (Webflow's details here come from its help centre), then read back what the custom domain serves. HubSpot sites that use its site search should add User-agent: HubSpotContentSearchBot with Allow: / before tightening anything.

Testing robots.txt for AI crawlers by fetching, not reading

Reading robots.txt tells you your intent. Only a request tells you your behaviour. Between an AI agent and your content there are four places it can be stopped, and only the first one is written in the file.

An AI agent asks for one of your pages1robots.txtDoes a group allowthis agent onthis path?2CDN / firewallDoes the edge return403 or a challengeto this agent?3ResponseSame status code asan ordinary browserrequest gets?4HTML bodyIs the text in theHTML, withoutrunning JavaScript?Check: read the live fileagainst your intentCheck: request the page once per agent, with a browser line as the control,and compare status codes and body sizes
Figure 3. Four checkpoints between an AI agent and your content. Only the first is written in robots.txt; the other three show up only when you request the page. Source: the test procedure described in this section.

Request one important page once per agent, with an ordinary browser user agent as a control, then fetch the robots.txt your live domain actually serves:

URL=https://example.com/pricing/
for ua in "Mozilla/5.0" GPTBot OAI-SearchBot ChatGPT-User \
          PerplexityBot Perplexity-User ClaudeBot Claude-SearchBot Claude-User; do
  printf "%-20s " "$ua"
  curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" -A "$ua" "$URL"
done

# And the file itself, exactly as your live domain serves it:
curl -s -w "\n-- %{http_code}\n" https://example.com/robots.txt

Read the output against this table:

What you seeWhat it meansWhere to fix it
The browser line fails tooThe site is down or the URL is wrong. Without this control, an outage looks exactly like a block.Fix the URL or the outage, then rerun.
403 on an agent you allowed, 200 for the browserThe edge is blocking it, whatever robots.txt says.CDN, firewall or bot-management rules, not the file.
200 for the agent but a far smaller body than the browser getsA failure, not a pass. Most AI crawlers do not execute JavaScript, so a script-built page reaches them nearly empty.Server rendering; see JavaScript rendering and AI crawlers.
Matching status and size across all linesThe server side is fine for this page.Read the served robots.txt against your intended position.

Two cautions keep this test honest. It shows what your server returns to each agent, not whether a crawler obeys your file, which is why the last command prints the file as actually served. And keep "could not fetch" separate from "not present": a timeout on robots.txt says nothing about whether the file exists.

A curl request from your own machine is also not the real crawler. An edge can recognise known agents by network origin as well as by name, so a mismatch is a lead to follow in your CDN's rules and logs, not a final verdict. If you would rather have these probes run with the browser control and the empty-body check built in, that is what InsightWonder's AI crawlability check does.

Sources