Every AI crawler, what it does, and what blocking it actually costs
17 user agents from 7 operators, each row taken from that vendor's own documentation and linked to it. Agents we could not verify against official documentation are left out rather than guessed, and the known gaps are listed at the bottom.
Grouped by what the agent is for, not by vendor
Vendors list their agents together, which is why one rule so often covers all of them. Grouping by purpose makes the decision separable: retrieval, user-triggered fetches, and training are three different questions.
Search / answer index
| User agent | Operator | What it does | What blocking it costs |
|---|---|---|---|
OAI-SearchBot | OpenAI | Builds the index that surfaces websites in ChatGPT’s search features. | Removal from ChatGPT search results. This is the agent sites most often block by accident while meaning to opt out of training. Vendor documentation · IP ranges |
PerplexityBot | Perplexity | Indexes pages so they can be surfaced and linked in Perplexity answers; the documentation states it is not used to crawl content for foundation models. | Removal from Perplexity answers. Perplexity cites sources on nearly every answer, so a block here shows up in citation data immediately. Vendor documentation |
Claude-SearchBot | Anthropic | Crawls to improve the relevance and accuracy of search results shown to Claude users. | Your pages stop informing Claude’s search results. Vendor documentation |
Google-CloudVertexBot | Crawls sites at the site owner’s own request when building Vertex AI Agents. | Only relevant if you are building a Vertex AI Agent on your own content. Vendor documentation | |
Applebot | Apple | Powers search technology across Apple experiences including Spotlight, Siri and Safari. | Removal from Apple’s search surfaces. Vendor documentation |
meta-webindexer | Meta | Analyses content to improve the quality of Meta AI search results. | Removal from Meta AI search results. Vendor documentation |
User-triggered fetch
| User agent | Operator | What it does | What blocking it costs |
|---|---|---|---|
ChatGPT-User may ignore robots.txt | OpenAI | Fetches a page because a person in ChatGPT or a Custom GPT asked for it. | A user who explicitly asks ChatGPT to read your page gets nothing back. Vendor documentation · IP ranges |
Perplexity-User may ignore robots.txt | Perplexity | Visits a page because a user’s question requires it, and links it in the response. | A user-requested visit fails. The documentation notes this agent generally does not follow robots.txt, because the request came from a person. Vendor documentation |
Claude-User may ignore robots.txt | Anthropic | Accesses a website because a Claude user asked a question that needs it. | A user-requested visit fails. Vendor documentation |
meta-externalfetcher may ignore robots.txt | Meta | Fetches links at a user’s request as part of agentic features. | A user-requested fetch fails; the documentation notes it may bypass robots.txt for user requests. Vendor documentation |
Training corpus
| User agent | Operator | What it does | What blocking it costs |
|---|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI’s generative foundation models. | Nothing in today’s answers. Blocking it is a defensible position on training use and does not remove you from ChatGPT search results. Vendor documentation · IP ranges |
ClaudeBot | Anthropic | Collects web content that may contribute to training Anthropic’s models. | Nothing in today’s answers; it is a training-use decision. Vendor documentation |
meta-externalagent | Meta | Crawls the web for uses such as training foundation models or indexing content directly. | Content stops feeding Meta’s models and direct indexing. Vendor documentation |
CCBot | Common Crawl | Builds an open, publicly available repository of web crawl data. | Your pages leave a widely redistributed open dataset. Common Crawl’s own page describes the repository as open to anyone rather than naming AI users, so treat downstream training use as likely but not vendor-confirmed. Vendor documentation |
Usage control (does not crawl)
| User agent | Operator | What it does | What blocking it costs |
|---|---|---|---|
Google-Extended | Not a crawler. It controls whether already-crawled content may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. | Removal from grounded Gemini answers, while Google Search inclusion and ranking are explicitly unaffected. This is the cleanest separation any vendor offers. Vendor documentation | |
Applebot-Extended | Apple | Not a crawler either. Disallowing it opts your content out of training Apple’s foundation models; the documentation states pages that disallow it can still appear in search results. | Nothing in search. It is purely a training-use decision. Vendor documentation |
Ad safety review
| User agent | Operator | What it does | What blocking it costs |
|---|---|---|---|
OAI-AdsBot | OpenAI | Checks the safety of pages submitted as ads in ChatGPT; visits only pages explicitly submitted. | Only relevant if you advertise in ChatGPT. Vendor documentation · IP ranges |
A robots.txt block that separates the three decisions
This allows the retrieval and user-triggered agents, and leaves the training opt-out commented out because that is your call, not ours. robots.txt is a declaration, not enforcement: verify with a real request afterwards.
# Retrieval agents: allowing these is what makes citation possible.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Applebot
Allow: /
User-agent: meta-webindexer
Allow: /
# User-triggered fetches: a person asked for this page by name.
User-agent: ChatGPT-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: meta-externalfetcher
Allow: /
# Training corpora are a separate decision. Uncomment to opt out.
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: meta-externalagent
# Disallow: /
# User-agent: CCBot
# Disallow: /
Sitemap: https://example.com/sitemap.xmlEngines with no agent you can allow by name
Absent from the table is not the same as absent from the web. These are the cases where we found no official documentation to cite.
Grok (xAI)
xAI does not publish a documented crawler user agent, so Grok cannot be allowed or blocked by name. What you can control is whether your pages are readable to general crawling at all.
Microsoft Copilot
We could not verify a Copilot-specific crawler in Microsoft’s public webmaster documentation, so no row is listed rather than a guessed one.
Questions teams ask before they start
What is the full list of AI crawlers?
This page lists 17 user agents from 7 operators, each taken from that vendor's own documentation and checked on 2026-09-09: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Google-CloudVertexBot, Applebot, Applebot-Extended, meta-externalagent, meta-webindexer, meta-externalfetcher, CCBot.
Which AI crawlers should I allow?
If you want to be citable in AI answers, allow the retrieval agents: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Google-CloudVertexBot, Applebot, meta-webindexer. Blocking one of these removes you from that engine's answers. The training crawlers (GPTBot, ClaudeBot, meta-externalagent, CCBot) are a separate decision with no effect on today's answers.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot gathers training content; OAI-SearchBot builds the index behind ChatGPT search. Blocking GPTBot is a training-use decision and does not remove you from ChatGPT search results. Blocking OAI-SearchBot does.
Do AI crawlers always obey robots.txt?
Not always, and the vendors say so themselves. Agents that fetch a page because a person asked for it are documented as possibly ignoring robots.txt, since the request came from a user rather than from automated crawling. Those rows are marked on this page.
Is Google-Extended a crawler?
No. Google-Extended and Applebot-Extended are usage controls, not crawlers: they govern whether already-crawled content may be used for AI training and grounding. Google documents that Google-Extended does not affect Search inclusion or ranking.
Every row links to the page it came from
- Checked on 2026-09-09 against each operator's own documentation. Vendors change these pages, so treat the date as part of the fact.
- Reading robots.txt does not prove access. A permissive file in front of a CDN rule or firewall that returns 403 to these agents is a block in practice, and the file will keep saying otherwise.
- Several agents are documented as possibly ignoring robots.txt when a person triggered the request; those rows are marked.
- Run the checker against your own site · Measure whether the answers cite you
Turn AI visibility into a repeatable growth system.
Create a project, confirm your market and start measuring the questions that matter.