Should You Block AI Crawlers? A Guide for B2B Websites

For most B2B websites the answer is: block selectively, not completely. AI crawlers fall into three groups. Search crawlers and user-triggered fetchers decide whether AI assistants can find and quote you today. Training crawlers only affect what future models learn. Blocking all of them to "protect content" usually just removes a manufacturer from the answers its buyers are reading.

Three kinds of AI crawler

Every major AI company now separates its bots by job. Once you know which job a bot does, the decision becomes much simpler.

Sources: OpenAI, Anthropic, Perplexity, Google.

Why B2B websites are different from publishers

News publishers block AI crawlers because their product is the article. If an AI summarizes the article, the publisher loses the visit and the ad revenue.

A manufacturer's website works the other way round. The product is the machine, not the page. Specifications, certifications and application notes exist so that buyers can find and compare suppliers. If an AI assistant reads those pages and recommends you, that is the page doing its job.

So the publisher's reasoning, "AI takes our content", rarely fits an exporter's catalog.

A sensible default for B2B websites

This is the setup we recommend as a starting point. It keeps you visible in AI search and leaves the training decision to you.

1. Allow search crawlers. OAI-SearchBot, Claude-SearchBot and PerplexityBot.
2. Don't block user-triggered fetchers. ChatGPT-User and Claude-User only visit because a person asked about something your page covers.
3. Make a deliberate choice on training crawlers. GPTBot, ClaudeBot and the Google-Extended token. Allowing them lets public product information become part of what future models know. Blocking them opts you out. Neither choice affects your Google rankings.
4. Keep private areas private. Block customer portals, internal search results and staging pages for every crawler, AI or not.

The exact robots.txt lines for each case are in robots.txt for AI Crawlers: Copy-Paste Examples.

What Google-Extended does and doesn't control

Google-Extended causes more confusion than any other name on the list.

According to Google's documentation, Google-Extended is not a separate crawler. It has no user agent of its own; Google crawls with its normal agents and reads the Google-Extended rule as an instruction. It controls whether content can be used for Gemini models and grounding.

Google also states that Google-Extended does not affect whether a site is included in Google Search or how it ranks. Blocking it is not a way to leave Google, and allowing it is not an SEO boost.

The bigger risk: blocking AI without meaning to

In our experience, the most common problem isn't a deliberate decision at all. It is a security layer quietly refusing AI requests:

• Security plugins on WordPress that block "unknown bots"
• Hosting firewalls that rate-limit or challenge crawlers
• CDN bot protection that treats AI agents as automated traffic

The robots.txt file may say "allow", while the server answers every AI request with 403 Forbidden. The site owner never sees it, because analytics tools don't record crawler visits.

The only way to know is to read the server log. How to Check Which AI Crawlers Visit Your Website walks through it.

A quick decision checklist

• Do you want to be recommended by ChatGPT, Claude or Perplexity? Allow their search crawlers.
• Is your content your product, like paid research or courses? Consider blocking training crawlers.
• Is your content marketing for a physical product? Blocking training usually brings little benefit.
• Have you checked that your firewall isn't blocking AI crawlers? If not, do that first.

FAQ

If I block GPTBot, will ChatGPT stop recommending my company?

Not necessarily. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking only GPTBot keeps you eligible for ChatGPT search results.

Does blocking AI crawlers improve my Google SEO?

No. AI crawler rules don't affect Google Search rankings. Google says Google-Extended is not a ranking signal and doesn't affect Search inclusion.

Do AI crawlers respect robots.txt?

The major companies say their automated crawlers do, including OpenAI, Anthropic and Perplexity. User-triggered fetchers such as ChatGPT-User may not apply robots.txt rules, because a person starts the request.

Can AI crawlers slow down my website?

Heavy crawling can add server load. Anthropic supports the Crawl-delay directive for its bots. If load is a concern, slowing a crawler is usually better than blocking it.