Short answer

AI companies do not visit your site with a single bot. This guide lists 24 bots from 11 companies, each verified against the company's own documentation. They do three different jobs: collecting data for model training, building the index behind AI search, and fetching a page at the moment a user asks about it. You can opt out of training and still be cited in AI search, as long as you know which name does what.

Below you will find the full table, two ready-to-copy robots.txt blocks, and the most common mistakes. Last verified: 11 November 2026. This page is updated at the same URL as the list changes.

AI crawler table (24 bots, 11 companies)

Every row is based on the company's own published documentation. "Type" shows why the bot visits; "Honors robots.txt?" reflects the company's own statement. Bots marked "Limited" act on a user's request, and their operators say openly that robots.txt rules may not apply.

Bot (robots.txt token)	Company	Type	Honors robots.txt?	What it does and what blocking means	Official source
`GPTBot`	OpenAI	Training	Yes	Crawls for model training. Blocking it keeps your content out of training.	OpenAI
`OAI-SearchBot`	OpenAI	Search / index	Yes	Feeds the ChatGPT search index. Blocking it removes you from ChatGPT search answers.	OpenAI
`ChatGPT-User`	OpenAI	User-triggered	Limited	Fetches a page when a user asks about it. OpenAI says robots.txt rules may not apply because the user initiates the request.	OpenAI
`ClaudeBot`	Anthropic	Training	Yes	Crawls for model training. Also supports the Crawl-delay directive.	Anthropic
`Claude-SearchBot`	Anthropic	Search / index	Yes	Indexes for Claude's search results. Blocking it reduces your visibility in Claude search.	Anthropic
`Claude-User`	Anthropic	User-triggered	Yes	Fetches pages when a user asks Claude a question. Blocking it stops Claude from retrieving your content for that query.	Anthropic
`Google-Extended`	Google	Control token	Yes	Not a separate bot but a control token. It decides whether content can be used for Gemini training and grounding. It does not affect Google Search ranking.	Google
`Google-CloudVertexBot`	Google	Search / index	Yes	Performs crawls that site owners request for Vertex AI agents. No effect on Google Search.	Google
`PerplexityBot`	Perplexity	Search / index	Yes	Crawls to surface and link websites in Perplexity search results.	Perplexity
`Perplexity-User`	Perplexity	User-triggered	Limited	Fetches pages on user request. Perplexity states plainly that it generally ignores robots.txt rules.	Perplexity
`Applebot`	Apple	Search / index	Yes	Crawls for search features such as Siri and Spotlight. Its data may also train Apple models unless the token below opts you out.	Apple
`Applebot-Extended`	Apple	Control token	Yes	Does not crawl; it only opts Applebot's data out of model training. You can still appear in search results.	Apple
`meta-externalagent`	Meta	Training	Yes	Crawls to train foundation AI models and improve products.	Meta
`meta-webindexer`	Meta	Search / index	Yes	Indexes content to improve Meta AI search results and cite sources in answers.	Meta
`meta-externalfetcher`	Meta	User-triggered	Limited	Fetches individual links a user requests. Meta says it may bypass robots.txt rules.	Meta
`Amazonbot`	Amazon	Training	Yes	Crawls to improve Amazon products; its data may train Amazon AI models. Ignores Crawl-delay.	Amazon
`Amzn-SearchBot`	Amazon	Search / index	Yes	Crawls for search experiences in Amazon products. Not used for generative model training.	Amazon
`Amzn-User`	Amazon	User-triggered	Limited	Fetches pages for user actions such as Alexa queries. Amazon says it may not follow all robots.txt directives.	Amazon
`CCBot`	Common Crawl	Training	Yes	Builds the non-profit open web archive. Because the archive is public, it feeds the training data of many language models.	Common Crawl
`MistralAI-Training`	Mistral	Training	Yes	Crawls to build training datasets for Mistral models.	Mistral
`MistralAI-Index`	Mistral	Search / index	Undocumented	Indexes for Mistral's search feature; not used for training. The documentation does not state its robots.txt behavior.	Mistral
`MistralAI-User`	Mistral	User-triggered	Yes	Fetches pages to answer a user's question. robots.txt governs which sites it may visit.	Mistral
`DuckAssistBot`	DuckDuckGo	Search / index	Yes	Fetches pages in real time for AI-assisted answers that cite sources. Data is not used for training; an opt-out takes effect within 72 hours and does not affect organic rankings.	DuckDuckGo
`Bytespider`	ByteDance	Training	Undocumented	The crawler of ByteDance, TikTok's owner. We found no official documentation of its behavior; if you need a firm block, enforce it at the server or CDN level.	None

Breakdown: 7 training bots, 9 search and index bots, 6 user-triggered bots, 2 control tokens.

Why the three types matter

Training bots (GPTBot, ClaudeBot, CCBot and others) collect content for language model training data. Blocking them does not remove you from AI search.
Search and index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) build the index AI assistants consult when answering. Block them and the assistant cannot cite you. We cover which assistant uses which index in a separate guide.
User-triggered bots (ChatGPT-User, Claude-User, Perplexity-User and others) do not crawl on their own; they arrive when someone asks an assistant about your page.

Ready-made robots.txt: no training, search allowed

This is the most requested setup: keep your content out of model training while staying citable in AI search. The block below disallows every training bot in the table plus the two tokens that control training (Google-Extended, Applebot-Extended). Search and user-triggered bots are untouched.

# AI training: disallowed. AI search and user fetches: allowed.
# Kaynak / Source: https://www.hititmedya.com/blog/ai-crawlers-list-user-agents-robots-txt (2026-11-11)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /

As a file: robots-ai-egitime-kapali.txt

Ready-made robots.txt: block all AI crawlers

This block disallows all 24 names in the table. The cost: ChatGPT, Claude, Perplexity and similar assistants will not be able to cite you.

# All documented AI crawlers: disallowed.
# Kaynak / Source: https://www.hititmedya.com/blog/ai-crawlers-list-user-agents-robots-txt (2026-11-11)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: Bytespider
Disallow: /

As a file: robots-ai-tumune-kapali.txt

To stay open to everything you do not need to add anything; if your robots.txt has no rule for these names, the bots may visit.

Seven common mistakes

Assuming that blocking Google-Extended affects Google Search. Google states that the token affects neither inclusion nor ranking in Search. Conversely, AI summaries in Search rely on normal Googlebot crawling; blocking Google-Extended does not remove you from them.
Looking for Google-Extended or Applebot-Extended in server logs. Neither sends requests of its own. They are control tokens that only mean something inside robots.txt.
Blocking search bots while trying to block training. GPTBot and OAI-SearchBot are different names, as are ClaudeBot and Claude-SearchBot. Blocking them all also erases your visibility in AI search.
Forgetting CCBot. The Common Crawl archive is public. Blocking individual company bots while leaving CCBot open largely defeats a training opt-out.
Treating robots.txt as enforcement. robots.txt is a request. Several user-triggered bots say in their own documentation that they may bypass it. A firm block needs a server or CDN rule.
Blocking outdated names. Many copy-paste lists include anthropic-ai, but Anthropic's current documentation names only three: ClaudeBot, Claude-SearchBot and Claude-User.
Forgetting subdomains. robots.txt applies per host. blog.example.com and www.example.com each need their own file.

Field data: 48 Turkish news sites

In November 2026 we examined the robots.txt files of 50 national news sites in Turkey; 48 could be measured. 10 sites block OpenAI's bots. In 7 of those 10 the AI section is identical, and that shared list lacks both CCBot and Google-Extended. These sites intend to opt out of training yet leave the two widest doors open, while also blocking search bots and giving up their visibility in AI search. Per-site results and the dataset: Turkish news sites and AI crawlers study.

Verifying that a bot is genuine

A user-agent string is trivial to fake. Not every request calling itself GPTBot comes from OpenAI. The reliable check is to compare the request's IP address against the list the company publishes:

OpenAI: openai.com/gptbot.json
Anthropic: claude.com/crawling/bots.json
Perplexity: www.perplexity.ai/perplexitybot.json
Common Crawl: index.commoncrawl.org/ccbot.json

OpenAI and Perplexity publish a separate list per bot; the addresses are on the official source pages linked in the table.

Beyond robots.txt: content usage signals

Listing bot names one by one does not scale; the list keeps changing. The industry is therefore moving toward signals that declare the purpose of use instead of the bot. Cloudflare's managed robots.txt adds a line of the form Content-Signal: search=yes, ai-train=no. Google's documentation also says it intends to support the shared AIPREF training preference signal in early 2027. Until such signals are widely honored, keeping the list above current is the safest approach.

Use this data

The full table is published in machine-readable form: ai-crawlers.json. Use it in your own tool, plugin or article. The license is CC BY 4.0: a link back to this page as the source is all we ask.

Sources

Every entry in the table was verified on 11 November 2026 against the following official documentation:

GPTBot

AI Crawlers List: robots.txt Guide for GPTBot, ClaudeBot and the Rest

GPTBot

AI Crawlers List: robots.txt Guide for GPTBot, ClaudeBot and the Rest