Short answer: AI crawler user agents do five different jobs, and the job, not the company behind the agent, decides what blocking it costs you. Training crawlers such as GPTBot and ClaudeBot cost you nothing in today's answers. Search agents such as OAI-SearchBot, PerplexityBot and Claude-SearchBot cost you the answers themselves. User-triggered fetchers such as ChatGPT-User cost a request a real person just made. Usage controls such as Google-Extended do not crawl at all; they decide what content that has already been crawled may be used for.
This article covers seventeen agents documented by OpenAI, Perplexity, Anthropic, Google, Apple, Meta and Common Crawl. Each row was checked against the vendor's own page on September 9, 2026, and every page is listed under Sources at the end. Where a vendor publishes no agent name, we say so rather than guess one, because a guessed name ends up in somebody's robots.txt doing nothing.
The five roles of AI crawler user agents
Read this table before the long one. Almost every robots.txt mistake comes from treating an operator as one decision when it is really several.
| Role | Agents | What blocking costs |
|---|---|---|
| Training corpus | GPTBot, ClaudeBot, meta-externalagent, CCBot | Nothing in today's answers. Declining training use is a defensible position. |
| Search and answer index | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, meta-webindexer, Google-CloudVertexBot | Removal from that engine's search results and answers. |
| User-triggered fetch | ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher | A request a real person made fails. These agents may not follow robots.txt at all. |
| Ad safety review | OAI-AdsBot | Only relevant if you advertise in ChatGPT. |
| Usage control (does not crawl) | Google-Extended, Applebot-Extended | Governs training and, for Google, grounding in Gemini. Search inclusion is documented as unaffected. |
Two rows carry most of the risk. OAI-SearchBot builds the index that surfaces websites in ChatGPT's search features, and it is the agent sites most often block by accident while meaning to opt out of training. Google-Extended is the opposite case: it looks like a crawler, but blocking it removes you from grounded Gemini answers while Google Search inclusion and ranking are explicitly unaffected. That is the cleanest separation any vendor offers, and it is easy to use without realising what it switches off.
Laid out by operator, the same information shows why "block OpenAI" or "allow Anthropic" is the wrong unit of decision. Most operators fill more than one column, and the columns have very different consequences.
Notice the gaps as well as the names. Perplexity documents no training crawler: its documentation states that PerplexityBot is not used to crawl content for foundation models. Google and Apple run their training controls as tokens rather than as separate crawlers. And two engines people ask about all the time, Grok and Copilot, have no row of named agents at all.
Every documented AI crawler user agent, one by one
The purpose column paraphrases each vendor's own description. The last column is the one a site owner actually needs.
| Agent | Operator | What it does | If you block it |
|---|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's generative foundation models. | No effect on today's answers, and it does not remove you from ChatGPT search results. |
OAI-SearchBot | OpenAI | Builds the index that surfaces websites in ChatGPT's search features. | Removal from ChatGPT search results. |
ChatGPT-User | OpenAI | Fetches a page because a person in ChatGPT or a Custom GPT asked for it. | A user who asks ChatGPT to read your page gets nothing back. |
OAI-AdsBot | OpenAI | Checks pages submitted as ChatGPT ads; visits only pages explicitly submitted. | Only matters for ChatGPT advertisers. |
PerplexityBot | Perplexity | Indexes pages so they can be surfaced and linked in Perplexity answers; documented as not used to crawl for foundation models. | Removal from Perplexity answers, visible in citation data almost at once because Perplexity cites sources on nearly every answer. |
Perplexity-User | Perplexity | Visits a page because a user's question needs it, and links it in the response. | A user-requested visit fails. Documented as generally not following robots.txt. |
ClaudeBot | Anthropic | Collects web content that may contribute to training Anthropic's models. | No effect on today's answers; a training-use decision. |
Claude-SearchBot | Anthropic | Crawls to improve the relevance and accuracy of search results shown to Claude users. | Your pages stop informing Claude's search results. |
Claude-User | Anthropic | Accesses a site because a Claude user asked a question that needs it. | A user-requested visit fails. |
Google-Extended | Not a crawler. Controls whether crawled content may train Gemini models and ground answers in Gemini Apps and Vertex AI. | Removal from grounded Gemini answers; Google Search unaffected. | |
Google-CloudVertexBot | Crawls a site at the owner's own request when building Vertex AI Agents. | Only relevant if you build a Vertex AI Agent on your own content. | |
Applebot | Apple | Powers search across Apple experiences including Spotlight, Siri and Safari. | Removal from Apple's search surfaces. |
Applebot-Extended | Apple | Not a crawler. Disallowing it opts content out of training Apple's foundation models. | Nothing in search; pages that disallow it can still appear in results. |
meta-externalagent | Meta | Crawls for uses such as training foundation models or indexing content directly. | Content stops feeding Meta's models and direct indexing. |
meta-webindexer | Meta | Analyses content to improve Meta AI search results. | Removal from Meta AI search results. |
meta-externalfetcher | Meta | Fetches links at a user's request as part of agentic features. | A user-requested fetch fails; documented as possibly bypassing robots.txt. |
CCBot | Common Crawl | Builds an open, publicly available repository of web crawl data. | Your pages leave a widely redistributed open dataset. Downstream training use is likely but not vendor-confirmed. |
One row deserves a second look. meta-externalagent is listed under training, but Meta's own description also mentions indexing content directly, so blocking it is a slightly wider decision than blocking GPTBot or ClaudeBot. If Meta AI search matters to you, the agent to keep allowed is meta-webindexer either way.
Classify an unfamiliar agent in three questions
New names appear in server logs all the time, and not every one of them is on a list like this. Before writing a rule, put the agent through the same three questions the table above answers, using the vendor's own crawler page rather than a forum post or a third-party bot directory.
The order matters. Ask first whether the name fetches anything, because a usage control such as Google-Extended never appears in your logs: the fetch is done by Googlebot or Applebot, and the token only decides what the content may be used for. Ask second whether a person triggered the request, because that is the group where robots.txt is weakest. OpenAI, Perplexity, Anthropic and Meta each document that their user-triggered agents may not follow it. Only then ask whether the agent feeds an index; if it does, a block is a decision to leave that engine's answers.
Applied to a whole operator rather than to a single agent, the difference between the two ways of writing rules looks like this:
| Approach | What the file says | What actually happens | When it is right |
|---|---|---|---|
| One rule per operator | Block every OpenAI agent | Training declined, ChatGPT search removed, user requests refused, all in one edit | Only if you genuinely want out of that product entirely |
| One rule per role | Block GPTBot, allow OAI-SearchBot, leave ChatGPT-User alone | Training declined; you stay eligible for ChatGPT search answers | The usual position for a business that wants to be recommended |
| Block by guessed name | A rule for an agent no vendor documents | Nothing; no crawler honours a name it does not send | Never. It only creates false confidence |
If your position is "no training, but stay eligible for answers", the file names the training agents and the search agents separately:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Google-Extended is left out deliberately. Disallowing it is not only a training decision, because it also removes you from grounded Gemini answers, so it deserves its own conscious choice; Gemini and Google-Extended walks through that trade-off. A full annotated file, including private paths that must be repeated in every named group, is in our robots.txt for AI crawlers, and the OpenAI case on its own is covered in how to block GPTBot without disappearing.
One vendor, three AI crawler user agents: the OpenAI case
OpenAI is the clearest example of why the role matters more than the name on the invoice. It runs three agents that a typical site sees, each with a separate trigger, a separate destination for your content and a separate published list of IP ranges.
Read the figure from left to right. GPTBot and OAI-SearchBot are both scheduled crawls run by OpenAI, so they look alike in a log file, yet they feed completely different things: one builds a possible training corpus, the other builds the index behind ChatGPT's search features. ChatGPT-User only arrives because someone in a conversation asked about your page, which is why blocking it fails a request a person made a moment ago rather than changing anything about training.
The separate IP lists are what make these agents verifiable. Any client can send the string "GPTBot", so the user agent alone proves nothing. Matching the source address in your logs against the list for the claimed agent turns a claim into evidence, and OpenAI publishes one for each of its agents as a JSON file.
What a list of AI crawler user agents cannot tell you
- Whether your server honours the rule. robots.txt states intent; only a fetch is evidence. A permissive file in front of a CDN or firewall rule that returns 403 to these agents is a block in practice, and the file will keep saying otherwise.
- Whether the agent reads robots.txt at all. User-triggered agents such as ChatGPT-User, Perplexity-User, Claude-User and meta-externalfetcher may not follow it, because a person made the request. Treat robots.txt as a statement to crawlers, not as access control.
- Whether the page is readable once fetched. Most AI crawlers do not execute JavaScript, so a 200 response with an empty shell is a failure, not a pass. JavaScript rendering and AI crawlers covers the common setups.
- Whether an undocumented agent exists. xAI does not publish a documented crawler user agent, so Grok cannot be allowed or blocked by name. We also could not verify a Copilot-specific crawler in Microsoft's public webmaster documentation.
The quickest check is to request one important page as each agent and as an ordinary browser, then compare status codes and response sizes. The browser request is the control: without it, a site that is simply down looks identical to one that blocks AI agents.
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "OAI-SearchBot" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$BROWSER_UA" https://example.com/
Keep "could not fetch" separate from "not there": a timeout on robots.txt says nothing about whether the file exists. And keep the list dated. Vendor documentation changes and new agents appear, so a crawler table without a checked-on date is a snapshot of an unknown moment. The maintained version of this table lives in the AI crawler list, and if you would rather have the agent-by-agent fetch run for you with the browser control built in, InsightWonder's AI crawlability check does exactly that.
Sources
- OpenAI: overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot), with IP lists such as gptbot.json
- Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User)
- Anthropic: how site owners can block the crawler (ClaudeBot, Claude-SearchBot, Claude-User)
- Google: common crawlers (Google-Extended, Google-CloudVertexBot)
- Apple: about Applebot (Applebot, Applebot-Extended)
- Meta: web crawlers (meta-externalagent, meta-webindexer, meta-externalfetcher)
- Common Crawl: CCBot