The AI Crawler Directory: GPTBot, ClaudeBot, PerplexityBot and More

This directory lists the AI crawlers and robots.txt control tokens you are most likely to meet, grouped by company. For each one it shows the job it does and whether it is a real crawler or only a rule name. Everything here comes from each company's own documentation, linked in every section.

How to read this list

AI agents fall into three groups, and the difference matters more than the company name:

• Search: indexes pages so an AI can cite them in answers.
• User-triggered: fetches a page because a person asked a question. These may not follow robots.txt, because a person starts the request.
• Training: collects content that may be used to train models.

Some names are control tokens, not crawlers. They never appear in your server logs. They only exist so you can write a robots.txt rule about how already-crawled content is used.

Summary table

OpenAI

Source: Overview of OpenAI Crawlers

• GPTBot crawls content that may be used to train OpenAI's generative AI models. Disallowing it opts your content out of training.
• OAI-SearchBot surfaces websites in ChatGPT's search features. This is the one to allow if you want to appear in ChatGPT answers.
• ChatGPT-User visits a page when a ChatGPT user, or a Custom GPT, needs it. It doesn't crawl automatically. OpenAI notes robots.txt rules may not apply.
• OAI-AdsBot checks landing pages submitted as ChatGPT ads. OpenAI says its data is not used to train foundation models.

OpenAI publishes an IP list for each agent, linked from the page above. Our guide What Is GPTBot? covers these agents in detail.

Anthropic

Source: Anthropic's crawler documentation

• ClaudeBot collects web content that could contribute to model training.
• Claude-SearchBot crawls to improve the quality of search results in Claude's answers.
• Claude-User fetches pages when a Claude user asks a question. Anthropic says blocking it can reduce your visibility in user-directed searches.

Anthropic's bots follow robots.txt and support the Crawl-delay directive. Anthropic also warns that blocking its IP addresses may stop its bots from reading your robots.txt.

Perplexity

Source: PerplexityBot documentation

• PerplexityBot collects pages to build Perplexity's search index. Perplexity says it follows robots.txt.

Perplexity's documentation notes that its IP ranges may change over time. Check its site for the current list before verifying requests.

Google

Source: List of Google's common crawlers

• Google-Extended is a control token, not a crawler. Google crawls with its normal user agents and reads Google-Extended rules to decide whether content can be used for Gemini models and grounding.

Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Google's search crawler remains Googlebot.

Apple

Source: About Applebot

• Applebot is Apple's crawler. Its data powers search features in Spotlight, Siri and Safari, and Apple says it may also help train Apple's foundation models.
• Applebot-Extended is a control token. Apple says it "does not crawl webpages". Disallowing it opts your content out of training Apple's foundation models, while your pages can still appear in Apple's search results.

Meta

Source: Meta Web Crawlers

• meta-externalagent crawls for purposes such as training AI models or indexing content. It can be blocked with robots.txt.
• meta-externalfetcher fetches individual links when a user asks for them. Meta says it may bypass robots.txt.
• facebookexternalhit reads shared links to build previews on Meta's apps. It is a link-preview fetcher rather than an AI crawler.

Common Crawl

Source: Common Crawl FAQ

• CCBot builds Common Crawl's open archive of the web. Its user agent is CCBot/2.0 (https://commoncrawl.org/faq/), and Common Crawl says it checks robots.txt before fetching pages.

Common Crawl's archive is public, so content it collects can be reused by many different organizations.

Keeping this list current

AI companies add, rename and retire agents regularly. Three habits help:

1. Review the official pages linked above every six months.
2. Match on the agent name, not the full user agent string. Version numbers change.
3. Verify by IP where the company publishes a list. User agent names are easy to fake.

To see which of these agents actually visit your site, follow How to Check Which AI Crawlers Visit Your Website.

FAQ

Which AI crawlers matter most for B2B websites?

For visibility in AI answers, the search and user-triggered agents matter most: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, and PerplexityBot. Training crawlers affect what future models learn.

Why can't I find Google-Extended or Applebot-Extended in my logs?

Both are control tokens, not crawlers. Google and Apple crawl with their normal user agents and read these tokens only as robots.txt instructions.

Do all AI crawlers follow robots.txt?

The automated crawlers listed here are documented as following robots.txt. Agents that act for a specific user, such as ChatGPT-User and meta-externalfetcher, may not, according to their operators.

How do I block all AI training crawlers at once?

There is no single rule. List each training agent or token, for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and meta-externalagent, in robots.txt. See robots.txt for AI Crawlers for examples.