GEOmonitorai Free audit
AI crawlers

What Is GPTBot? How OpenAI's Crawlers Read Your Website

By the GEOmonitorai team ·

GPTBot is OpenAI's crawler for collecting content that may be used to train its models. It is one of three OpenAI agents you will see in your server logs. The other two, OAI-SearchBot and ChatGPT-User, decide whether your pages can show up in ChatGPT's answers today. Treating all three the same is the most common mistake we see on B2B websites.

The three OpenAI agents at a glance

OpenAI publishes a separate name, purpose and IP list for each agent. They are controlled separately in robots.txt, so you can allow one and block another.

Agent What it does Follows robots.txt Official IP list
GPTBot Crawls content that may be used to train OpenAI's generative AI models Yes gptbot.json
OAI-SearchBot Finds and indexes pages so they can appear in ChatGPT search results Yes searchbot.json
ChatGPT-User Fetches a page when a ChatGPT user's question needs it; does not crawl on its own Possibly not, because a user starts the request chatgpt-user.json

Source: OpenAI's overview of its crawlers. OpenAI also runs OAI-AdsBot for checking ad landing pages, which most B2B sites will rarely see.

GPTBot: training

GPTBot visits pages that may become part of the data used to train future models. Blocking it tells OpenAI not to use your content for training.

What blocking GPTBot does not do, according to OpenAI's documentation: it does not remove you from ChatGPT search. That is handled by OAI-SearchBot.

A typical GPTBot user agent contains:

compatible; GPTBot/1.4; +https://openai.com/gptbot

The version number changes over time, so match on the name GPTBot rather than the full string.

OAI-SearchBot: ChatGPT search

When a buyer asks ChatGPT "Which suppliers make 200-ton hydraulic presses?", ChatGPT search looks at pages that OAI-SearchBot has already discovered. If your robots.txt blocks OAI-SearchBot, you are opting out of ChatGPT search. OpenAI notes that opted-out sites may still appear as plain navigational links.

For most manufacturers and exporters, OAI-SearchBot is the agent you most want to allow. It is the closest thing to "Googlebot for ChatGPT."

ChatGPT-User: a real person is asking

ChatGPT-User visits a page only when someone in ChatGPT, or in a Custom GPT, asks something that needs that page. It does not browse the web automatically.

That makes it the most direct signal in your logs: each visit means a real person's question led ChatGPT to your page. If you sell to other businesses, a ChatGPT-User request for one of your product pages is close to a buyer reading your catalog.

Because these requests are started by users, OpenAI notes that robots.txt rules may not apply to them.

How to see OpenAI's crawlers in your logs

Google Analytics and similar tools will not show any of these agents. They count visits with a browser script, and crawlers don't run scripts. The only reliable record is your web server's access log.

On a typical Nginx server, this shows recent OpenAI requests:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User" /var/log/nginx/access.log | tail -20

Two things to check in the results:

  1. The status code. 200 means the page was served. 403 means something, often a firewall or security plugin, refused the request. 404 means the crawler asked for a page that doesn't exist.
  2. The IP address. Anyone can put "GPTBot" in a user agent. Compare the IP against OpenAI's published lists linked above before you trust it.

We cover the full process, including other AI platforms, in How to Check Which AI Crawlers Visit Your Website.

Should you allow GPTBot?

It depends on what you want from AI. For a B2B website that wants to be found, a common setup is:

  • Allow OAI-SearchBot. This affects whether ChatGPT search can use your pages now.
  • Don't block ChatGPT-User. These are visits driven by real questions.
  • Decide on GPTBot. Allowing it lets your public product information be part of what future models learn. Blocking it opts you out of training. Neither choice changes your Google rankings.

If you choose to block only training, the robots.txt rule is:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

For complete examples covering Anthropic, Perplexity and Google, see robots.txt for AI Crawlers: Copy-Paste Examples.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

Not by itself. According to OpenAI, GPTBot is used for training data, while ChatGPT search relies on OAI-SearchBot. To stay visible in ChatGPT search, allow OAI-SearchBot.

Does blocking GPTBot affect Google rankings?

No. GPTBot is OpenAI's crawler. Google Search uses Googlebot, and a robots.txt rule for GPTBot has no effect on it.

How do I know a GPTBot visit is real?

Check the request's IP address against the list OpenAI publishes at openai.com/gptbot.json. Requests that use the GPTBot name from other IPs are not from OpenAI.

Why don't I see GPTBot in Google Analytics?

Analytics tools count visits with JavaScript, and crawlers don't run it. Crawler visits only appear in server or CDN logs.

See which AI crawlers read your site

GEOmonitorai reads your server logs and shows which AI platforms visit, which key pages they never reach, and where requests get blocked. The first audit is free.

Get a free AI visibility audit

Related reading