The AI crawler you block by accident

Every AI company runs at least two different crawlers, and they do opposite jobs. One collects pages that might train a future model. The other fetches your page at the moment somebody asks a question, so the assistant can cite it. They have different names in your robots.txt. Blocking the first is a legitimate business decision. Blocking the second removes you from the answer entirely, and almost nobody who does it meant to.

This is the most common way we find a site excluded from AI answers, and it is rarely a deliberate decision. It is usually one line added during a "should we let AI train on our content?" conversation that nobody revisited, or a blanket firewall rule applied to anything with "bot" in the name.

The two kinds of crawler

The distinction matters more than any other technical detail in this area, so it is worth being precise about which bot is which.

Training crawlers collect content that may be used to build future models. Blocking them does not remove you from anything a user sees today. It removes your content from a training set. Whether that is worth doing is a real question about your content and your business, and reasonable companies answer it differently.

Retrieval crawlers fetch pages live, when a question is asked, so the assistant can read and cite them. Blocking these does exactly one thing: it makes you uncitable. There is no upside. Your content is not protected from anything, because it was never going into a training set via that route — it simply stops being available to quote.

Here is what each company actually documents, with the tokens exactly as they appear.

Company	Training	Retrieval or user-initiated
OpenAI	GPTBot	OAI-SearchBot, ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot, Claude-User
Perplexity	(none — see below)	PerplexityBot, Perplexity-User
Google	Google-Extended	Googlebot

A few details from the official documentation that are easy to get wrong:

• OpenAI's crawler documentation describes OAI-SearchBot as being "used to surface websites in search results in ChatGPT's search features", and ChatGPT-User as being for "certain user actions in ChatGPT" — explicitly "not used for crawling the web in an automatic fashion". Only GPTBot is described as training-related.
• Anthropic documents three: ClaudeBot for model development, Claude-SearchBot to improve search result quality, and Claude-User for when a person asks Claude something that requires fetching a page.
• Perplexity states that neither of its bots is used for training foundation models. PerplexityBot indexes for search visibility and Perplexity-User fetches on a user action. If you block either, you have given up citations and protected nothing.

The Google-Extended misunderstanding

This one costs people real visibility, so it deserves its own heading.

A lot of sites added Google-Extended to their robots.txt believing it opts them out of AI Overviews. It does not. Google's own crawler documentation describes it as a token publishers use to manage whether content "may be used for training future generations of Gemini models", and states plainly that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Google's AI features in Search are served from the ordinary Search index, reached by ordinary Googlebot. So:

• Blocking Google-Extended does not remove you from AI Overviews.
• The only way to stay out of them is to be out of Google Search, which is a much larger decision than most people are making when they add that line.

If you blocked Google-Extended to avoid AI Overviews, it did not work, and it did cost you a say in Gemini training. Worth knowing either way.

How to check yours in two minutes

Open yoursite.com/robots.txt and read it. You are looking for three things.

One: any Disallow: / under a retrieval bot. Search the file for OAI-SearchBot, PerplexityBot, Claude-User, Claude-SearchBot, ChatGPT-User and Perplexity-User. If any of them has a blanket disallow, that is your answer and it is a one-line fix.

Two: a blanket block on anything bot-shaped. Some security tools and CDNs block unfamiliar user-agents at the edge, before robots.txt is ever consulted. A clean robots.txt does not prove a crawler can reach you — check your CDN or firewall rules too. This is the version of the problem that does not show up in any SEO tool, because the tool reads your robots.txt and sees nothing wrong.

Three: bots that were not named at all. If your file only lists User-agent: * with Allow: /, everything is permitted and you are fine. Naming crawlers explicitly is a way of making the decision visible to whoever edits the file next, not a requirement.

We name ours explicitly for that reason. Writing this article, we noticed our own list predates Anthropic publishing Claude-SearchBot and OpenAI publishing their ads validation bot — neither is blocked, because the wildcard rule permits them, but the list is a document as well as a directive and ours had drifted.

What blocking actually protects

It is worth being honest about the trade, because "block everything" is sometimes framed as the cautious choice and it is not.

Blocking training crawlers protects your content from being absorbed into a model's weights. If you publish something genuinely proprietary — original research, a large reference corpus, paid content — that can be a defensible call.

Blocking retrieval crawlers protects nothing. The page is public. A person can read it, quote it and paraphrase it. All you have prevented is an assistant naming you as the source while it describes your category to a buyer.

That asymmetry is the whole decision. Most businesses should allow retrieval unconditionally and think carefully, once, about training.

Access is necessary, not sufficient

Fixing your robots.txt does not get you cited. It removes a reason you cannot be. Assistants overwhelmingly cite pages that already rank well and describe entities they can identify confidently, so the work that follows is conventional: pages that answer real questions, consistent naming across the web, and presence on the third-party sources these systems lean on.

If access is not your problem — the crawlers can reach you and you are still not being named — the reasons are different, and we have gone through why an assistant names your competitor instead of you separately.

That is the substance of AI search visibility, and it sits on top of ordinary technical SEO rather than replacing it — the same way site speed is not a growth strategy but will quietly cap one.

If you want someone to check whether the assistants can currently reach and identify you, tell us your site and we will tell you what we find, including when the answer is that nothing is wrong.