What Is ClaudeBot? Anthropic's Crawlers Explained

ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train its Claude models. It is one of three Anthropic agents that can appear in your server logs. The other two, Claude-SearchBot and Claude-User, affect whether Claude can find and read your pages when someone asks it a question. A B2B website that blocks "Claude" with one rule may lose more visibility than it intended.

The three Anthropic agents at a glance

Anthropic documents each agent separately in its crawler help article, and each one has its own robots.txt name.

Agent	What it does	If you block it
ClaudeBot	Collects public web content that may contribute to model training	Signals that your future content should be excluded from Anthropic's training data
Claude-SearchBot	Crawls to improve the relevance and accuracy of Claude's search results	Your pages are not indexed for Claude's search, which may reduce your visibility in its answers
Claude-User	Fetches a page when a Claude user asks a question that needs it	Claude cannot retrieve your page in response to that user's question

[Documented] Anthropic says its bots honor robots.txt and do not bypass CAPTCHAs. For ClaudeBot, it also supports the non-standard Crawl-delay directive, so you can slow it down instead of blocking it.

This structure mirrors OpenAI's, where GPTBot trains models and OAI-SearchBot powers search. See What Is GPTBot? for the comparison.

How to spot Anthropic's crawlers in your logs

Each agent includes its name in the user agent string, so a search of your access log finds them:

grep -Ei "ClaudeBot|Claude-SearchBot|Claude-User" /var/log/nginx/access.log | tail -20

A name in the user agent is not proof. Anyone can send a request that claims to be ClaudeBot. [Documented] Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json, and says a request from an IP on that list comes from Anthropic. Check suspicious traffic against the list before you draw conclusions or write firewall rules.

[Analysis] What to look for in the results:

Status codes. 200 means the page was delivered. 403 or 503 usually means a firewall, CDN or security plugin refused the request, even if robots.txt allows it. Our guide on firewalls blocking AI crawlers explains how to find the cause.
Which pages. Are product, application and company pages being read, or only the homepage and images?
Claude-User visits. Each one means a real person asked Claude something that led it to your page. They are worth watching even when the numbers are small.

robots.txt examples

To block only training and keep Claude's search and user requests:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

To slow ClaudeBot down instead of blocking it:

User-agent: ClaudeBot
Crawl-delay: 10

[Documented] Anthropic says these rules must be in the robots.txt file at the top level of each subdomain you want them to apply to. A rule on www.example.com does not cover shop.example.com.

Anthropic also advises against blocking its IP addresses as a way to opt out, because IP-based methods "may not work correctly" over time. Use robots.txt instead.

For ready-made setups covering OpenAI, Anthropic, Perplexity and Google together, see robots.txt for AI crawlers. Before you publish changes, paste your file into our free AI crawler robots.txt checker to confirm each agent gets the result you want.

The new AI training signal

[Documented] As of October 30, 2026, Anthropic says it recognizes a machine-readable "no AI training" preference based on the IETF AIPREF draft standard, and notes the standard may still change. If your robots.txt has a group that names ClaudeBot, that rule takes priority over the general training signal for the same path.

[Analysis] For most B2B sites, this changes little in practice. If you already control ClaudeBot by name, those rules still decide. If you want one training preference for every AI provider, the new signal is worth following. VALTARIQ explains the signal in AI training control for websites.

Should a B2B website allow ClaudeBot?

[Analysis] Search and user agents are the easy decision. Claude-SearchBot and Claude-User are how Claude finds and reads your pages when a buyer asks about suppliers, so most exporters and manufacturers should allow both.

ClaudeBot is a business choice. Your public product pages, specifications and application notes are written to be read and repeated. Allowing training may help AI models describe your products and company correctly. If some content is valuable precisely because it isn't freely available, such as detailed design guides or paid documentation, keep it behind a login or block that directory. Our guide on whether to block AI crawlers walks through the trade-offs.

Whatever you decide, check your logs afterwards. A robots.txt rule only shows what you asked for. Your logs show what each crawler actually did.

FAQ

Does blocking ClaudeBot remove my site from Claude?

Not by itself. According to Anthropic, ClaudeBot collects training data, while Claude-SearchBot and Claude-User handle search and user requests. To stay visible when people ask Claude, allow those two.

Does ClaudeBot follow robots.txt?

Yes. Anthropic says its crawlers honor robots.txt directives, and ClaudeBot also supports Crawl-delay.

How do I verify a ClaudeBot request is real?

Compare the request's IP address with the list Anthropic publishes at claude.com/crawling/bots.json. A request that uses the ClaudeBot name from another IP did not come from Anthropic.

Why do I see ClaudeBot in my logs but not in Google Analytics?

Crawlers don't run JavaScript, so analytics tools never count them. Crawler visits only appear in server or CDN logs.