Is Your Firewall Blocking AI Search? How to Find Out

Your robots.txt can welcome every AI crawler while your server quietly turns them away. Security plugins, hosting firewalls and CDN bot protection are built to stop automated traffic, and AI crawlers are automated traffic. The result is a site that looks open on paper but returns 403 Forbidden to ChatGPT and Claude. Here is how to find out, layer by layer.

Why this happens without anyone noticing

Three things make this problem easy to miss:

• Analytics can't see it. Crawlers don't run JavaScript, so blocked crawler requests never reach your analytics dashboard.
• The site works for people. You, your team and your customers browse normally.
• Defaults change. A plugin update, a new CDN rule or a hosting migration can switch on stricter bot rules overnight.

The only reliable evidence is in your server or CDN logs.

Step 1: confirm there is a problem

Count the status codes AI crawlers receive:

grep -E "OAI-SearchBot|ChatGPT-User|GPTBot|ClaudeBot|Claude-SearchBot|PerplexityBot" \
  /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

If there are no AI requests in your origin logs at all, check whether a CDN sits in front of your site. A CDN can block requests before they ever reach your server.

Step 2: check each layer

Requests pass through several layers. Any of them can block AI crawlers.

CDN or proxy

If you use Cloudflare or another CDN, look at its security events or firewall logs, filtered by user agent. Bot protection features may challenge or block AI crawlers. Many CDNs now offer separate settings for AI crawlers; check what yours is set to.

Hosting firewall and server panel

Control panels such as cPanel, Plesk and aaPanel often include a web application firewall or "anti-crawler" setting. Look for rules that block unknown user agents or limit request rates.

Security plugins

WordPress security plugins commonly include bot blocking, country blocking and rate limiting. Check their block logs for AI agent names.

Web server rules

Search your Nginx or Apache configuration for rules that match user agents:

grep -rniE "gptbot|claudebot|bot|crawler|spider" /etc/nginx/ /etc/apache2/ 2>/dev/null

Old "block bad bots" lists sometimes include AI crawlers.

Step 3: fix it without opening the door to bad bots

You don't need to switch protection off. Make a targeted exception instead:

1. Allow by verified identity, not by name alone. Anyone can claim to be GPTBot. Where possible, allow the IP ranges each company publishes, such as OpenAI's lists at openai.com/gptbot.json and openai.com/searchbot.json.
2. Use your CDN's verified bot features if it offers them, instead of writing your own rules.
3. Prefer rate limits to blocks if crawler load is the concern. Anthropic also supports Crawl-delay in robots.txt.
4. Keep robots.txt as your policy. Decide what to allow there. See robots.txt for AI Crawlers.

Avoid blocking AI companies by IP as a way to "opt out". Anthropic notes this can stop its bots from reading robots.txt at all, so your other rules can't be applied.

Step 4: verify the fix

After the change, give it a few days and run the Step 1 command again. You should see:

• the share of 403 and 429 responses fall
• 200 responses rise for the same agents
• requests reaching product pages, not only the homepage

AI crawlers don't visit every day, so a single quiet day doesn't mean the fix failed. Two to four weeks of data give a clearer picture.

FAQ

Can a firewall block AI crawlers even if robots.txt allows them?

Yes. robots.txt is a set of instructions that crawlers read. Firewalls act on the request itself and can refuse it regardless of what robots.txt says.

Is it safe to allow AI crawlers through my firewall?

It is generally safe to allow verified requests from the published IP ranges of major AI companies. Allowing every request that only claims an AI name is riskier, because names are easy to fake.

Will unblocking AI crawlers slow my site down?

For most B2B sites, AI crawler traffic is small compared with normal visitors. If load becomes a problem, rate-limit the crawler rather than blocking it.

How do I check if Cloudflare blocks AI bots?

Look at Cloudflare's security events filtered by the bot's user agent, and review its bot and AI crawler settings for your domain. Cloudflare's own logs show requests that never reached your server.