How to Check Which AI Crawlers Visit Your Website

AI crawlers leave no trace in Google Analytics, but every request they make is written to your web server's access log. In about ten minutes you can find out which AI platforms visit your site, which pages they read, and whether your server is turning them away. This guide shows how, from a single command to continuous monitoring.

Why your analytics shows nothing

Google Analytics, Matomo and similar tools count a visit when a small JavaScript file runs in the visitor's browser. AI crawlers request your HTML directly and don't run that script.

The result: you can have hundreds of AI requests a week and see zero of them in your dashboard. To see them you need a record made by the server itself, the access log.

Step 1: find your access log

If several websites share one log file, separate them first, or you'll mix up their traffic.

Step 2: search for AI user agents

On Linux, this counts requests from the main AI crawlers in today's log:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

Example output:

    214 OAI-SearchBot
     88 GPTBot
     41 ClaudeBot
     12 ChatGPT-User
      3 PerplexityBot

These numbers are illustrative. Yours will depend on your site and how long the log covers.

Google-Extended won't appear here. It is a robots.txt control token, not a crawler with its own user agent.

Step 3: see which pages they read

Counting visits is only the start. The useful question is what they read:

grep "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

This lists the 20 paths OAI-SearchBot requested most. Compare the list with your most important pages. A common finding is that crawlers spend their visits on the homepage, tag pages and old URLs, while key product pages get no visits at all.

Step 4: check the status codes

The status code tells you whether the crawler actually got the page:

grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

A high share of 403 responses is the most important thing to look for. It means robots.txt may say "allow" while your security layer says "no".

Step 5: verify the crawler is real

Anyone can write "GPTBot" in a user agent. Before you draw conclusions, check the IP addresses:

• OpenAI publishes IP lists for each agent: GPTBot, OAI-SearchBot, ChatGPT-User.
• Perplexity points to its published IP information in its crawler documentation.
• Google crawlers can be verified with a reverse DNS lookup, as described in Google's crawler documentation.

Requests that claim an AI name but come from other IPs should be counted separately as unverified.

Doing this every week

Running commands by hand works for a one-off check. It gets hard when you want to track trends, compare months or watch several websites. You need to:

• keep logs long enough (many servers rotate them after 7–14 days)
• refresh IP lists as companies change them
• compare crawled pages against your sitemap
• notice when 403 errors suddenly rise

This is the work GEOmonitorai automates. A small collector on your server sends access logs to a dashboard, which identifies and verifies AI crawlers, shows which key pages they never reach, and flags blocked requests.

FAQ

Can I see AI crawlers in Google Analytics 4?

No. GA4 records visits through JavaScript, which crawlers don't run. You need server logs or CDN logs.

My site is on Shopify or Wix. Can I check AI crawlers?

Not from raw server logs, because these platforms don't expose them. You can still check robots.txt, and test AI answers directly by asking ChatGPT or Perplexity about your products.

How often do AI crawlers visit?

It varies a lot by site. Small B2B sites may see a few AI requests a day; others see thousands. That's why a few weeks of data give a more reliable picture than a single day.

What is the difference between ChatGPT-User and OAI-SearchBot in my logs?

OAI-SearchBot crawls on its own to build ChatGPT's search index. ChatGPT-User visits only when a person's question needs your page. A ChatGPT-User visit is a direct sign that someone asked about something on your site.