GEOmonitorai Free audit
GEO Basics

Your Website May Be Blocking AI: A robots.txt and Firewall Check

By the GEOmonitorai team ·

A website can block AI in two ways: robots.txt tells crawlers not to crawl, or a firewall, CDN or security plugin refuses the request outright. The first you can see by opening one file. The second only shows up in server logs. Many companies have a perfectly good robots.txt and still fail on the second.

The first door: robots.txt

robots.txt is a text file in the root of your website that tells crawlers where they may and may not go. Open "yourdomain.com/robots.txt" in a browser to see it.

Three common mistakes:

  1. The "do not index" setting from development was never removed. The site had Disallow: / before launch and nobody changed it back, so no crawler can get in.
  2. Blocking every AI crawler. Worried that "AI will copy our content", the site blocks GPTBot, ClaudeBot and PerplexityBot, and the search crawlers go with them.
  3. Mixing up training and search. The goal was only to keep content out of model training, but the crawler that decides whether you appear in AI search was blocked too.

For most export manufacturers, product pages exist to be seen by buyers, and the more the better. A sensible default is: allow all AI search and user-triggered crawlers; decide for yourself whether to allow training crawlers.

For example, to keep content out of OpenAI's training while staying in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Before you change anything, paste your robots.txt into GEOmonitorai's free robots.txt checker. It shows whether each of 14 major AI crawlers is allowed or blocked. The tool runs in your browser and doesn't upload your content.

The second door: firewalls, CDNs and security plugins

robots.txt is only a request, which well-behaved crawlers follow. A firewall simply refuses, whatever robots.txt says.

These settings most often block AI crawlers by mistake:

Source What happens
CDN bot protection With "block AI bots" or strict bot protection turned on, AI crawlers get a 403 or a challenge page
WordPress security plugins Frequent crawler visits are treated as an attack and whole IP ranges are banned
Server rate limits A crawler requests too many pages at once and receives 429 or 503
Country blocking Only some countries are allowed, while most AI crawlers run from US servers

What these problems have in common: the site works perfectly in a browser and nothing warns you. You only find out when the logs show rows of GPTBot or ClaudeBot requests getting 403.

A three-step check

  1. Open your robots.txt and use the checker to confirm AI search crawlers are allowed.
  2. Ask your web host or developer whether your CDN or security plugin has an option like "block AI crawlers" turned on.
  3. Look at the last two weeks of server logs and count how many AI crawler requests returned 200 and how many returned 403, 429 or 503.

You can do the first two yourself. The third is a standard item in every GEOmonitorai audit report, which lists the AI platform, the pages and the times of each block.

Further reading: robots.txt for AI Crawlers: Copy-Paste Examples · Is Your Firewall Blocking AI Search? How to Find Out · Free AI crawler robots.txt checker

GEO Basics series, part 5 of 10. Next: Why AI Gets Your Company Wrong: Build a Fact Base

See which AI crawlers read your site

GEOmonitorai reads your server logs and shows which AI platforms visit, which key pages they never reach, and where requests get blocked. The first audit is free.

Get a free AI visibility audit

Related reading