AI Crawler List: User Agents and How to Allow or Block Them
AI companies use several different crawlers, and they don’t all do the same job. Some collect training data. Some build search indexes. Some fetch a page only when a person asks a question. Blocking the wrong one can quietly remove your site from AI answers.
This page lists the main AI crawlers, what each one does, and the robots.txt rules to control them.
Three kinds of AI crawlers
- Training crawlers collect public pages that may be used to train future AI models. Blocking them does not remove you from AI search.
- Search crawlers build the index that AI search features use to find and cite pages. Blocking them can make you invisible in AI search results.
- User-triggered fetchers visit a page when someone asks an AI assistant about it. Blocking them stops the assistant from reading your page on request.
The list
| Company | User agent token | Type | What it does |
|---|---|---|---|
| OpenAI | GPTBot | Training | Collects data for training OpenAI models |
| OpenAI | OAI-SearchBot | Search | Indexes pages for ChatGPT search |
| OpenAI | ChatGPT-User | User-triggered | Fetches pages when a ChatGPT user asks |
| Anthropic | ClaudeBot | Training | Collects data that may be used to train Claude models |
| Anthropic | Claude-SearchBot | Search | Indexes pages for Claude’s search answers |
| Anthropic | Claude-User | User-triggered | Fetches pages when a Claude user asks |
| Perplexity | PerplexityBot | Search | Indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | User-triggered | Fetches pages when a Perplexity user asks |
Google-Extended | Training (control token) | Not a separate crawler. Controls whether content Google crawls is used to train and ground Gemini models. Does not affect inclusion or ranking in Google Search | |
| Apple | Applebot-Extended | Training (control token) | Not a separate crawler. Controls whether content crawled by Applebot is used to train Apple’s generative AI models |
| Microsoft | Bingbot | Search | Bing’s standard crawler for Bing search |
| Meta | Meta-ExternalAgent | Training | Crawls for uses such as training Meta’s AI models and indexing content |
| Meta | Meta-ExternalFetcher | User-triggered | Fetches individual links when a Meta AI user asks. Meta notes it may bypass robots.txt |
| Amazon | Amazonbot | Training | Improves Amazon products and services; may be used to train Amazon AI models |
| Amazon | Amzn-SearchBot | Search | Indexes pages for search experiences in Amazon products such as Alexa |
| Amazon | Amzn-User | User-triggered | Fetches live information for user requests, such as Alexa questions |
| Common Crawl | CCBot | Training (open dataset) | Builds a public web archive widely used to train AI models |
| DuckDuckGo | DuckAssistBot | User-triggered | Fetches pages in real time for DuckDuckGo’s AI-assisted answers |
| Mistral | MistralAI-User | User-triggered | Fetches pages when a user asks Mistral’s Vibe assistant |
Each token is controlled separately. Blocking ClaudeBot, for example, does not block Claude-SearchBot or Claude-User.
Common robots.txt setups
Allow AI search, block AI training
robots.txt
# Block AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: CCBot
Disallow: /
# Allow AI search and user requests
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /Allow all AI crawlers
robots.txt
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
User-agent: CCBot
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: MistralAI-User
Allow: /
Sitemap: https://example.com/sitemap.xmlBlock all AI crawlers
robots.txt
# Bingbot is not listed: blocking it also removes you from Bing search.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: Amzn-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: DuckAssistBot
Disallow: /
User-agent: MistralAI-User
Disallow: /Note: this also removes your site from AI search answers.
Robots.txt is not the whole story
A robots.txt “Allow” only works if the crawler can actually reach your server. Check these too:
- CDN and firewall settings. Some CDNs block AI crawlers by default on new sites. Check your CDN’s bot or AI crawler settings.
- Bot protection challenges. A CAPTCHA or JavaScript challenge page stops crawlers even when robots.txt allows them.
- JavaScript rendering. Most AI crawlers don’t run JavaScript. If your main content loads through scripts, they may see an empty page.
How to check if a visit is really from an AI crawler
Anyone can fake a user agent. To confirm a visit is genuine, compare the request’s IP address with the IP ranges each company publishes, or use reverse DNS where the company supports it.
Check your own site
Gainwide will test all of this for you – permissions, access and readability – in one check.
Sources
- OpenAI: Overview of OpenAI Crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity Crawlers
- Google: Google’s common crawlers
- Apple: About Applebot
- Microsoft: Which crawlers does Bing use?
- Meta: Meta Web Crawlers
- Amazon: Amazon Web Search crawlers
- Common Crawl: CCBot
- DuckDuckGo: About DuckAssistBot
- Mistral: Mistral robots