Short answer: Your robots.txt should name AI bots by job, not treat them as one group. Allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited in ChatGPT, Claude and Perplexity answers.
Blocking the training bots (GPTBot, ClaudeBot, Google-Extended) is a separate, legitimate choice that does not remove you from those answers. And check the CDN in front of your site, because a firewall rule that returns 403 to a bot means it never reads robots.txt at all.
Why is there more than one AI bot per company?
Each vendor now documents separate user agents for separate jobs. That is the whole reason a blanket 'block the AI bots' rule goes wrong: it lumps together a bot that trains a model, a bot that builds a search index, and a bot that fetches a page because someone asked about it five seconds ago.
We checked each vendor's own documentation on 2026-11-13, and the table below is what they say. It is a snapshot. Crawlers change, so treat the vendor pages in the sources as the authority when they disagree with us.
| Operator | User agent | Job | Blocking it means |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Not shown in ChatGPT search answers (can still appear as plain links) |
| OpenAI | GPTBot | Crawls content that may train models | Opt-out of training |
| OpenAI | ChatGPT-User | Fetches a page when a user asks | OpenAI says robots.txt rules may not apply |
| Anthropic | ClaudeBot | Collects content that may contribute to training | Opt-out of training |
| Anthropic | Claude-SearchBot | Indexes content to improve search results | May reduce visibility in Claude search answers |
| Anthropic | Claude-User | Fetches when a user asks Claude a question | May reduce visibility for user-directed search |
| Perplexity | PerplexityBot | Surfaces sites in Perplexity; not used for model training | Not shown in Perplexity results |
| Perplexity | Perplexity-User | Fetches when a user asks | Perplexity says it generally ignores robots.txt |
| Googlebot | Search, and therefore AI Overviews and AI Mode | Leaves Google Search entirely | |
| Google-Extended | Product token only, no separate fetch; controls Gemini training and grounding in Gemini Apps | Does not affect Search inclusion or ranking | |
| Apple | Applebot-Extended | Opt-out token for training Apple foundation models | Opt-out of training; Applebot can still crawl |
Should I block GPTBot?
That depends on a decision only you can make: are you comfortable with your content being used to train models? Blocking GPTBot or ClaudeBot is a policy choice, not a defect, and plenty of publishers make it on purpose. What it does not do is remove you from ChatGPT or Claude answers, because those draw on the separate search bots.
So there are three sensible postures, and none of them is wrong. The mistake is landing on one by accident.
- Be cited, training is up to you: Allow the search and user-fetch bots. Whether to allow the training bots is a separate call, and allowing them is the default if you never touch robots.txt.
- Be cited, but opt out of training: Block GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
- Stay out of AI entirely: Block all of the above. Understand that this removes you from AI citations, and confirm that is the intent before shipping it.
When a failing audit is not a problem
Our audit's aeo.ai-bots-allowed rule fails when robots.txt disallows the whole site for any of GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended. That list mixes training bots with search bots, so a site that blocks only the training bots on purpose will fail it. If that is you, the report is blunt, not you. Keep the choice and ignore the finding.
The aeo.ai-bots-allowed audit rule fails if robots.txt disallows the whole site for any one of five user agents: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended. A site with no robots.txt blocks nothing, so the rule passes. (AEO Toolkit skill source)
What do people get wrong about robots.txt and AI?
- Blocking Google-Extended does not remove you from AI Overviews: Google says Googlebot is the control for AI Overviews and AI Mode. Google-Extended only governs Gemini training and grounding in Gemini Apps. Limiting what Google shows takes snippet controls such as nosnippet or max-snippet, or noindex.
- A Disallow will not stop user-fetch bots: ChatGPT-User and Perplexity-User act for a person who just asked a question. OpenAI and Perplexity both say robots.txt may not govern them. A hit from one is a person asking, not a rule being broken.
- Changes are not instant: OpenAI and Perplexity both say robots.txt changes can take up to about 24 hours to take effect. Anthropic gives no number. Allow a day before you decide the fix did not work.
- An empty Disallow means the opposite of Disallow: /: Disallow with no value allows everything. Disallow: / blocks everything. One character, opposite meanings.
- A staging rule that shipped: User-agent: * with Disallow: / is a normal staging setting. It is also the single most common reason a live site is invisible. Check the live file after every deploy.
Why can my CDN block AI bots even if robots.txt allows them?
robots.txt is a request. A firewall rule is enforcement. If a bot gets a 403 or a challenge page at the edge, it never gets as far as reading your robots.txt, so an Allow line in the file changes nothing. This is the most common reason a site that 'allows everything' still cannot be read.
The usual sources are Cloudflare's AI crawler policies, which can add managed Disallow lines and block AI training crawlers at the edge, a custom WAF rule or Bot Fight Mode that challenges unfamiliar user agents, and security plugins on WordPress. Both layers have to agree. Allowing a bot in robots.txt while a CDN policy is set to Block is a contradiction the CDN wins.
If you want a bot through a firewall, allow it by user agent plus its published IP range, not by user agent alone, because user agents are trivially forged. OpenAI, Perplexity and Anthropic each publish where their crawlers come from.
A quick test, and why it can mislead you
Request your homepage with a bot's user agent and compare the status code to a normal browser request. A 403, 429 or challenge page for the bot is a hint that something at the edge is blocking it. But a CDN that verifies bots by IP range will let the real bot through while blocking your spoofed request, so a 200 proves little. Server logs and your CDN's bot analytics are the real evidence.
How do I check and fix my AI crawler access?
At summary level it is five steps. The exact robots.txt syntax for your framework lives in the implementation guide linked below.
Read your live robots.txt
Open yourdomain.com/robots.txt. It should return 200 as plain text, not an HTML page. A catch-all route that returns your app shell for this path counts as no robots.txt.
Decide your posture
Pick one of the three above. If you are unsure, the first one, allowing the search bots, is the one that keeps you eligible for citation.
Write explicit groups for the bots you named
A group that names a bot replaces the wildcard group for that bot, so repeat any Disallow paths you still want. Reference your sitemap at the bottom.
Check the layer in front of your server
Look at your CDN's AI bot policy, any WAF rules, and any security plugin. Make sure they match what robots.txt says.
Verify after a day
Re-read the live file, then look at your CDN analytics or server logs for 403 and 429 responses to the search bots over the last few days. No blocks there is the real pass.
This is exactly the kind of fix a coding agent can do for you. The skill behind this article can be installed into Claude Code and other agents, so the agent reads your robots.txt, checks the edge, and proposes the change. You decide the posture; it writes the file.
Full implementation guide for AI crawler accessThe source of the skill is open, and the page below explains how to install skills into your agent.
Install the AEO agent skillsSkill source on GitHub, if you want to read what it does before you let an agent run it.
aeo-ai-crawler-access on GitHubWhat will fixing this not do?
Letting a bot in is necessary, not sufficient. It makes your pages fetchable. It does not make an assistant choose them, and nobody can promise that. If your pages are also empty until JavaScript runs, or are not in the search index, a welcoming robots.txt will not help, and those are separate problems with separate fixes.
One more caveat on measurement. Our AI visibility tools check Perplexity only. They cannot tell you whether ChatGPT, Claude or Gemini cite you, so do not use them as proof that a robots.txt change worked on those engines. Ask the engines your buyer questions directly and record who gets named.
Common questions
Should I block GPTBot in robots.txt?
Only if you do not want your content used to train OpenAI models. GPTBot is the training crawler. Blocking it does not remove you from ChatGPT search, which uses a separate bot called OAI-SearchBot. It is a policy choice, not a mistake, and you can make it while staying eligible for citation.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces sites in ChatGPT search. Blocking OAI-SearchBot means your site is not shown in ChatGPT search answers, although it can still appear as a plain link. They are controlled separately in robots.txt.
Does blocking Google-Extended remove me from Google AI Overviews?
No. Google-Extended only controls use of your content for Gemini training and for grounding in Gemini Apps, and it does not affect Search inclusion or ranking. Google says Googlebot is the control for AI Overviews and AI Mode, and snippet controls or noindex limit what is shown.
Why are AI bots blocked when my robots.txt allows them?
Most often a CDN or firewall rule is returning 403 or a challenge page before the bot ever reads robots.txt. Cloudflare's AI crawler policies, a custom WAF rule, Bot Fight Mode and WordPress security plugins are the usual causes. Allow the bots by user agent plus published IP range, and make both layers agree.
Sources
Related guides
Want it done for you
Want this done and re-checked every month? Our monthly maintenance re-runs your buying questions in ChatGPT, Perplexity and Google, ships the fixes that keep you out of the answers, and reports what moved. Or run the free check on your site first at advancelabs.dev/tools/audit.
See how it works