AI Crawlers List: robots.txt Guide for GPTBot, ClaudeBot and the Rest
Short answer
AI companies do not visit your site with a single bot. This guide lists 24 bots from 11 companies, each verified against the company's own documentation. They do three different jobs: collecting data for model training, building the index behind AI search, and fetching a page at the moment a user asks about it. You can opt out of training and still be cited in AI search, as long as you know which name does what.
Below you will find the full table, two ready-to-copy robots.txt blocks, and the most common mistakes. Last verified: 11 November 2026. This page is updated at the same URL as the list changes.
AI crawler table (24 bots, 11 companies)
Every row is based on the company's own published documentation. "Type" shows why the bot visits; "Honors robots.txt?" reflects the company's own statement. Bots marked "Limited" act on a user's request, and their operators say openly that robots.txt rules may not apply.
| Bot (robots.txt token) | Company | Type | Honors robots.txt? | What it does and what blocking means | Official source |
|---|---|---|---|---|---|
GPTBot | OpenAI | Training | Yes | Crawls for model training. Blocking it keeps your content out of training. | OpenAI |
OAI-SearchBot | OpenAI | Search / index | Yes | Feeds the ChatGPT search index. Blocking it removes you from ChatGPT search answers. | OpenAI |
ChatGPT-User | OpenAI | User-triggered | Limited | Fetches a page when a user asks about it. OpenAI says robots.txt rules may not apply because the user initiates the request. | OpenAI |
ClaudeBot | Anthropic | Training | Yes | Crawls for model training. Also supports the Crawl-delay directive. | Anthropic |
Claude-SearchBot | Anthropic | Search / index | Yes | Indexes for Claude's search results. Blocking it reduces your visibility in Claude search. | Anthropic |
Claude-User | Anthropic | User-triggered | Yes | Fetches pages when a user asks Claude a question. Blocking it stops Claude from retrieving your content for that query. | Anthropic |
Google-Extended | Control token | Yes | Not a separate bot but a control token. It decides whether content can be used for Gemini training and grounding. It does not affect Google Search ranking. | ||
Google-CloudVertexBot | Search / index | Yes | Performs crawls that site owners request for Vertex AI agents. No effect on Google Search. | ||
PerplexityBot | Perplexity | Search / index | Yes | Crawls to surface and link websites in Perplexity search results. | Perplexity |
Perplexity-User | Perplexity | User-triggered | Limited | Fetches pages on user request. Perplexity states plainly that it generally ignores robots.txt rules. | Perplexity |
Applebot | Apple | Search / index | Yes | Crawls for search features such as Siri and Spotlight. Its data may also train Apple models unless the token below opts you out. | Apple |
Applebot-Extended | Apple | Control token | Yes | Does not crawl; it only opts Applebot's data out of model training. You can still appear in search results. | Apple |
meta-externalagent | Meta | Training | Yes | Crawls to train foundation AI models and improve products. | Meta |
meta-webindexer | Meta | Search / index | Yes | Indexes content to improve Meta AI search results and cite sources in answers. | Meta |
meta-externalfetcher | Meta | User-triggered | Limited | Fetches individual links a user requests. Meta says it may bypass robots.txt rules. | Meta |
Amazonbot | Amazon | Training | Yes | Crawls to improve Amazon products; its data may train Amazon AI models. Ignores Crawl-delay. | Amazon |
Amzn-SearchBot | Amazon | Search / index | Yes | Crawls for search experiences in Amazon products. Not used for generative model training. | Amazon |
Amzn-User | Amazon | User-triggered | Limited | Fetches pages for user actions such as Alexa queries. Amazon says it may not follow all robots.txt directives. | Amazon |
CCBot | Common Crawl | Training | Yes | Builds the non-profit open web archive. Because the archive is public, it feeds the training data of many language models. | Common Crawl |
MistralAI-Training | Mistral | Training | Yes | Crawls to build training datasets for Mistral models. | Mistral |
MistralAI-Index | Mistral | Search / index | Undocumented | Indexes for Mistral's search feature; not used for training. The documentation does not state its robots.txt behavior. | Mistral |
MistralAI-User | Mistral | User-triggered | Yes | Fetches pages to answer a user's question. robots.txt governs which sites it may visit. | Mistral |
DuckAssistBot | DuckDuckGo | Search / index | Yes | Fetches pages in real time for AI-assisted answers that cite sources. Data is not used for training; an opt-out takes effect within 72 hours and does not affect organic rankings. | DuckDuckGo |
Bytespider | ByteDance | Training | Undocumented | The crawler of ByteDance, TikTok's owner. We found no official documentation of its behavior; if you need a firm block, enforce it at the server or CDN level. | None |
Breakdown: 7 training bots, 9 search and index bots, 6 user-triggered bots, 2 control tokens.
Why the three types matter
- Training bots (GPTBot, ClaudeBot, CCBot and others) collect content for language model training data. Blocking them does not remove you from AI search.
- Search and index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) build the index AI assistants consult when answering. Block them and the assistant cannot cite you. We cover which assistant uses which index in a separate guide.
- User-triggered bots (ChatGPT-User, Claude-User, Perplexity-User and others) do not crawl on their own; they arrive when someone asks an assistant about your page.
Ready-made robots.txt: no training, search allowed
This is the most requested setup: keep your content out of model training while staying citable in AI search. The block below disallows every training bot in the table plus the two tokens that control training (Google-Extended, Applebot-Extended). Search and user-triggered bots are untouched.
# AI training: disallowed. AI search and user fetches: allowed.
# Kaynak / Source: https://www.hititmedya.com/blog/ai-crawlers-list-user-agents-robots-txt (2026-11-11)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /
As a file: robots-ai-egitime-kapali.txt
Ready-made robots.txt: block all AI crawlers
This block disallows all 24 names in the table. The cost: ChatGPT, Claude, Perplexity and similar assistants will not be able to cite you.
# All documented AI crawlers: disallowed.
# Kaynak / Source: https://www.hititmedya.com/blog/ai-crawlers-list-user-agents-robots-txt (2026-11-11)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: Bytespider
Disallow: /
As a file: robots-ai-tumune-kapali.txt
To stay open to everything you do not need to add anything; if your robots.txt has no rule for these names, the bots may visit.
Seven common mistakes
- Assuming that blocking Google-Extended affects Google Search. Google states that the token affects neither inclusion nor ranking in Search. Conversely, AI summaries in Search rely on normal Googlebot crawling; blocking Google-Extended does not remove you from them.
- Looking for Google-Extended or Applebot-Extended in server logs. Neither sends requests of its own. They are control tokens that only mean something inside robots.txt.
- Blocking search bots while trying to block training. GPTBot and OAI-SearchBot are different names, as are ClaudeBot and Claude-SearchBot. Blocking them all also erases your visibility in AI search.
- Forgetting CCBot. The Common Crawl archive is public. Blocking individual company bots while leaving CCBot open largely defeats a training opt-out.
- Treating robots.txt as enforcement. robots.txt is a request. Several user-triggered bots say in their own documentation that they may bypass it. A firm block needs a server or CDN rule.
- Blocking outdated names. Many copy-paste lists include
anthropic-ai, but Anthropic's current documentation names only three: ClaudeBot, Claude-SearchBot and Claude-User. - Forgetting subdomains. robots.txt applies per host.
blog.example.comandwww.example.comeach need their own file.
Field data: 48 Turkish news sites
In November 2026 we examined the robots.txt files of 50 national news sites in Turkey; 48 could be measured. 10 sites block OpenAI's bots. In 7 of those 10 the AI section is identical, and that shared list lacks both CCBot and Google-Extended. These sites intend to opt out of training yet leave the two widest doors open, while also blocking search bots and giving up their visibility in AI search. Per-site results and the dataset: Turkish news sites and AI crawlers study.
Verifying that a bot is genuine
A user-agent string is trivial to fake. Not every request calling itself GPTBot comes from OpenAI. The reliable check is to compare the request's IP address against the list the company publishes:
- OpenAI: openai.com/gptbot.json
- Anthropic: claude.com/crawling/bots.json
- Perplexity: www.perplexity.ai/perplexitybot.json
- Common Crawl: index.commoncrawl.org/ccbot.json
OpenAI and Perplexity publish a separate list per bot; the addresses are on the official source pages linked in the table.
Beyond robots.txt: content usage signals
Listing bot names one by one does not scale; the list keeps changing. The industry is therefore moving toward signals that declare the purpose of use instead of the bot. Cloudflare's managed robots.txt adds a line of the form Content-Signal: search=yes, ai-train=no. Google's documentation also says it intends to support the shared AIPREF training preference signal in early 2027. Until such signals are widely honored, keeping the list above current is the safest approach.
Use this data
The full table is published in machine-readable form: ai-crawlers.json. Use it in your own tool, plugin or article. The license is CC BY 4.0: a link back to this page as the source is all we ask.
Sources
Every entry in the table was verified on 11 November 2026 against the following official documentation: