robots.txt for AI Crawlers: Copy-Paste Examples

This page gives four robots.txt setups you can copy, from "allow everything" to "allow AI search but opt out of training". Each one uses the user agent names that OpenAI, Anthropic, Perplexity and Google publish. Before you paste anything, read the section on common mistakes: one misplaced line can hide your whole site from AI search.

Where robots.txt lives and how it works

robots.txt is a plain text file at the root of your domain, for example https://www.example.com/robots.txt. Each subdomain needs its own file.

A crawler looks for the group that names it with User-agent, then follows the Allow and Disallow lines in that group. If no group names it, it follows the User-agent: * group.

Two rules people often miss:

• A crawler uses only its most specific group. If you write a GPTBot group, GPTBot ignores everything under User-agent: *.
• robots.txt is a request, not a lock. The major AI companies say they follow it. It does not stop anyone determined to ignore it.

The AI user agents you need to know

Sources: OpenAI, Anthropic, Perplexity, Google. Companies add and rename agents, so check these pages once or twice a year.

Example 1: allow all AI crawlers

If your robots.txt only contains a general group like this, every AI crawler above may crawl the whole site:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

You don't need to list AI crawlers individually to allow them. Adding a Sitemap line helps every crawler find your pages.

Example 2: allow AI search, opt out of training

This is the most common choice for B2B websites that want to be found in AI answers but don't want their content used for model training:

# AI search and user-triggered fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# Training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Listing several User-agent lines above one set of rules applies those rules to all of them.

Example 3: allow AI, but keep private areas closed

Most sites have sections no crawler should see. Repeat the private paths in each group, because each crawler reads only its own group:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /cart/
Disallow: /search

User-agent: *
Allow: /
Disallow: /account/
Disallow: /cart/
Disallow: /search

Example 4: slow a crawler down instead of blocking it

If an AI crawler puts noticeable load on your server, slowing it down keeps you visible. Anthropic supports the Crawl-delay directive for its bots:

User-agent: ClaudeBot
Crawl-delay: 5

Crawl-delay is not part of the official robots.txt standard, and not every crawler supports it. Check each company's documentation.

Five mistakes that block AI by accident

1. A leftover Disallow: / under User-agent: *. Often left over from a staging site. It blocks every crawler that doesn't have its own group, AI included.
2. A specific group that forgets the basics. You add User-agent: GPTBot with one Disallow line, and GPTBot no longer sees the rules under *. Repeat anything important.
3. Blocking Google-Extended to "leave AI". It controls Gemini training and grounding, not Google Search. It won't remove you from Google, and it won't improve your rankings either.
4. Blocking by IP instead of robots.txt. Anthropic notes that blocking its IP addresses may stop its bots from reading your robots.txt at all, so your other rules can't be applied.
5. Trusting robots.txt alone. The file can say "allow" while a firewall answers AI crawlers with 403. Only your server logs show what actually happened.

How to test your robots.txt

• Open https://yourdomain.com/robots.txt in a browser and read it line by line.
• In Google Search Console, the robots.txt report shows the version Google last fetched and any parsing errors.
• Check your server logs after a few days. If an AI crawler you allowed still shows 403 errors, the block is somewhere else. See How to Check Which AI Crawlers Visit Your Website.

FAQ

Do I need to list every AI crawler to allow it?

No. A crawler with no group of its own follows the User-agent: * rules. You only need named groups when you want different rules for different crawlers.

Is user agent matching case-sensitive?

Under the robots.txt standard, user agent names are matched case-insensitively. Using the exact spelling each company publishes is still the safest habit.

How long until a robots.txt change takes effect?

Crawlers cache robots.txt. Google, for example, generally caches it for up to 24 hours. Other crawlers use their own schedules.

Should I block AI crawlers from my product pages?

For most manufacturers and exporters, no. Product pages are exactly what buyers ask AI assistants about. Blocking search crawlers there makes it harder for AI answers to mention you.