AI crawler robots.txt checker
Paste your robots.txt and see which AI crawlers it allows or blocks: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. Everything runs in your browser; nothing is uploaded.
Open https://yourdomain.com/robots.txt in a new tab, copy everything, and paste it here. The box starts with an example.
Results for /
| Agent | Type | Result | Why |
|---|
How this checker works
It follows the matching rules of the robots.txt standard (RFC 9309): a crawler uses the group that names it, or the User-agent: * group if none does. Within that group the longest matching rule wins, and Allow wins a tie. * and $ wildcards are supported.
Agent names come from each company's documentation. See our AI crawler directory for what each one does, and robots.txt for AI crawlers for ready-made setups.
What it can't tell you
robots.txt is a set of instructions. A firewall, security plugin or CDN can still refuse AI crawlers with 403 errors, even when robots.txt allows them. Agents that act for a user, such as ChatGPT-User, may not follow robots.txt at all. Only your server logs show what really happened. Here is how to check them.
Want to know what AI crawlers actually did?
GEOmonitorai reads your server logs and shows which AI platforms visited, which pages they read, and which requests were blocked.
Get a free AI visibility audit