AI Crawlers Visit Your Website Every Day. Why Can't You See Them in Analytics?
AI crawlers are programs that AI companies send to read web pages. They don't run tracking code, so Google Analytics and similar tools can't see them at all. To know whether AI has visited your website and what it read, you have to look at your server or CDN access logs.
Three types of AI crawler
The major AI companies split their crawlers into several types, each with its own name. Knowing the difference tells you which ones to allow.
| Type | What it does | Examples |
|---|---|---|
| Search crawlers | Build the AI search index that decides whether you appear in AI search results | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot |
| User-triggered | Someone asks an AI a question and it opens your page on the spot | ChatGPT-User, Claude-User, Perplexity-User |
| Training crawlers | Collect public content that may be used to train models | GPTBot (OpenAI), ClaudeBot (Anthropic) |
Sources: OpenAI crawler documentation, Anthropic crawler documentation.
One more is easy to miss: Google's AI Overviews and AI Mode use the normal search index, crawled by Googlebot. Google-Extended is a control token that decides whether content may be used for models such as Gemini.
Why analytics tools can't see them
Analytics tools work like this: a piece of JavaScript sits in the page, the visitor's browser runs it, and it sends a record of the visit back to the analytics server.
AI crawlers only download the page content and don't run that code. So even if GPTBot reads hundreds of your pages a day, GA4 shows nothing. And if a firewall blocks AI crawlers, analytics won't warn you either.
Your server, on the other hand, writes a line in its access log for every request: the time, the page, the status code, the requester's name (User-Agent) and IP address. Every AI crawler visit is there.
What the logs show
A log line looks roughly like this (simplified):
2026-11-03 14:21 GET /products/screw-air-compressor/ 200 OAI-SearchBot
2026-11-03 14:25 GET /products/screw-air-compressor/ 403 ClaudeBot
A few weeks of logs answer four questions:
- Which AI platforms visited, how often, and whether visits are rising or falling.
- Which pages they read. Were core product pages and the company page read, or only the homepage and images?
- Were requests blocked? 403 and 503 responses often mean a firewall or security plugin is blocking AI.
- Was it really an AI crawler? Anyone can use the GPTBot name. OpenAI and Anthropic both publish their crawler IP lists; only checking the IP confirms a visit is genuine.
Checking yourself vs using a tool
If you have server access, a technician can get a first look with one command:
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot" access.log | tail -50
Manual checks have three problems: logs are often deleted after a few days, trends are hard to compare over time, and fake crawlers are hard to spot.
GEOmonitorai installs a read-only log collector on your server that continuously sends access records to the monitoring platform. It identifies AI crawlers, verifies them and flags blocked requests. IP addresses are masked before storage. The collector doesn't change your website and can be stopped and removed at any time.
Further reading: How to Check Which AI Crawlers Visit Your Website · The AI Crawler Directory: GPTBot, ClaudeBot, PerplexityBot and More · Free AI crawler robots.txt checker
GEO Basics series, part 4 of 10. Next: Is Your Website Blocking AI? robots.txt and Firewall Check
See which AI crawlers read your site
GEOmonitorai reads your server logs and shows which AI platforms visit, which key pages they never reach, and where requests get blocked. The first audit is free.
Get a free AI visibility audit