# GPTBot, ClaudeBot and Every Other AI Crawler
URL: https://doableclaw.com/blog/gptbot-and-ai-crawlers/
> GPTBot, ClaudeBot, OAI-SearchBot and the rest — what each is for, which ones fire when a person is actually asking, and the two that never appear in your log.
Published: 2026-09-03

> **TL;DR:** Every major vendor now runs three kinds of agent — one collecting training data, one building a search index, and one that fires because a person is asking right now. Sorting your log into those three buckets changes what it says. Two of the most-discussed names, Google-Extended and Applebot-Extended, are not crawlers and will never appear. Blocking one agent does not block its siblings, blocking all of them does not stop your content being cited through somebody else's copy of it, and every name in the file is a claim until you check the IP against the vendor's published ranges.

You grepped your access log, and now you have a list of names. `GPTBot`. `ClaudeBot`. `Meta-ExternalAgent`. Something called `OAI-SearchBot` that you have never heard of.

They are not interchangeable, and the differences decide what you can do about any of them. Some are collecting training data. Some are building a search index. Some fired because a person was asking a question at that exact moment. Two of the names most commonly discussed are not crawlers at all and will never appear in your log.

Every row below is taken from the operator's own published documentation, linked, and checked on **2 September 2026**. This roster changes — vendors add agents, rename them, and split one into three. If you are reading this much later, follow the links rather than the table.

## The Quick Answer

- [OpenAI alone runs four separate agents, each with its own robots.txt token](#the-roster)
- [An undocumented name is unknown, not an AI engine](#the-roster)
- [Sort every name into three buckets: training, index, and live](#three-way-split)
- [The index crawlers are the closest thing in your log to being in the running](#three-way-split)
- [Counting "AI bot hits" as one number destroys the only distinction that matters](#three-way-split)
- [Google-Extended and Applebot-Extended never make a request, so grep finds nothing](#not-crawlers)
- [Blocking GPTBot opts you out of training and leaves ChatGPT's search index untouched](#blocking)
- [31% of domains blocking GPTBot were still cited, via syndication and forum quotes](#blocking)
- [The live fetchers sit outside robots.txt on purpose, in each vendor's own words](#blocking)
- [A user agent is a claim — match the IP against the vendor's published range file](#verify)

## Which AI bots are these? The full list

Here is every documented agent from the major vendors, grouped by who runs it. The column that matters most is **when it fires**, because that is what separates a bot indexing you in the background from a bot fetching your page because somebody asked about you thirty seconds ago.

| Vendor | User agent | When it fires | robots.txt |
|---|---|---|---|
| **OpenAI** | `GPTBot` | training data collection | obeys |
| | `OAI-SearchBot` | building ChatGPT's search index | obeys |
| | `ChatGPT-User` | a person asked, right now | *may not apply* |
| | `OAI-AdsBot` | validating a submitted ad page | obeys |
| **Anthropic** | `ClaudeBot` | training data collection | obeys |
| | `Claude-SearchBot` | improving search result quality | obeys |
| | `Claude-User` | a person asked, right now | obeys |
| **Perplexity** | `PerplexityBot` | building the search index | obeys |
| | `Perplexity-User` | a person asked, right now | *generally ignores* |
| **Google** | `Googlebot` | Search, Images, News, Discover | obeys |
| | `GoogleOther` | generic, various product teams | obeys |
| | `Google-CloudVertexBot` | crawls a site owner requested for Vertex AI | obeys |
| **Meta** | `meta-externalagent` | training foundation models | obeys |
| | `meta-webindexer` | Meta AI search result quality | obeys |
| | `meta-externalfetcher` | a person asked, right now | *may bypass* |
| | `meta-externalads` | advertising products | obeys |
| | `facebookexternalhit` | someone shared your link | *may bypass* |
| **Apple** | `Applebot` | powers Apple search features | obeys |
| **Amazon** | `Amazonbot` | products, services, and Amazon AI training | obeys |
| **Common Crawl** | `CCBot/2.0` | open crawl archive, used by many | obeys |

Sources, in order: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/guides/bots), [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), [Apple](https://support.apple.com/en-us/119829), [Amazon](https://developer.amazon.com/amazonbot), [Common Crawl](https://commoncrawl.org/ccbot).

Names you will see that are not on this list are not necessarily fake — plenty of smaller engines and SEO tools crawl too, and some are documented nowhere useful. Treat an undocumented name as unknown rather than as an AI engine.

## Why every vendor now runs three different bots

Because three different things are happening, and lumping them together is what makes a log unreadable. Read down the table and the same shape repeats at OpenAI, Anthropic, Perplexity and Meta: **a training crawler, a search-index crawler, and a live fetcher that runs when a user asks a question.**

```
training        GPTBot · ClaudeBot · meta-externalagent
index           OAI-SearchBot · Claude-SearchBot · PerplexityBot · meta-webindexer
live            ChatGPT-User · Claude-User · Perplexity-User · meta-externalfetcher
```

That middle row is the one worth caring about. Training crawls feed a model that ships months later. The index row is what a chat engine actually searches when it needs a current answer, so it is the closest thing in your log to being in the running for a citation.

The bottom row is closer still, and it is the most misread. A live fetch means somebody typed a question and the engine went and got your page to help answer it. It is the strongest single signal in the file. It is also not proof you were named in the answer it produced — [a bot visiting is not the same as being mentioned](/blog/check-ai-traffic-free/), and the gap between the two is the thing that makes logs feel more informative than they are.

One practical consequence: **counting "AI bot hits" as a single number destroys the only distinction that matters.** A thousand training crawls and one live fetch is a very different week from the reverse, and a total tells you which one you had.

## The two names you will never find in your log

`Google-Extended` and `Applebot-Extended` are not crawlers. They are robots.txt control tokens — labels you use to grant or refuse permission — and neither one ever makes a request. Grep for them forever and you will find nothing, because there is nothing to find.

Both vendors say so themselves.

| Token | What it actually controls | The vendor's own words |
|---|---|---|
| `Google-Extended` | whether content Google **already crawled** may train future Gemini models | "doesn't have a separate HTTP request user agent string" |
| `Applebot-Extended` | how the data `Applebot` **already collected** gets used | "does not crawl webpages" — and disallowed pages "can still be included in search results" |

This trips people up in both directions. Blocking Google-Extended does not reduce your crawl traffic by a single request, because it was never making any. And an empty grep for these names is not evidence that Google or Apple are ignoring you — it is the expected result, always, for everybody.

The general rule: **the permission layer and the request layer are different systems.**

- **The request layer** — a bot fetches your page, and your server writes a line about it. This is the only one that leaves a trace.
- **The permission layer** — what may be done with your content after it has been collected. It never makes a request, so it never appears anywhere you can grep.

---

## Does blocking GPTBot stop ChatGPT from citing me?

No — and OpenAI's own documentation is the fastest way to see why. `GPTBot` is described as the crawler for "content that may be used in training our generative AI foundation models." It is one of four OpenAI agents. `OAI-SearchBot` is the one "used to surface websites in search results in ChatGPT's search features," and it is governed by its own token.

So a robots.txt that disallows `GPTBot` and nothing else opts you out of training while leaving ChatGPT's search index entirely untouched. Many sites that believe they have blocked OpenAI have blocked exactly one of its four agents, and not the one that feeds answers.

Write `Disallow: GPTBot` and you have opted out of training data collection. Three OpenAI agents are still running:

| Still running | What it does |
|---|---|
| `OAI-SearchBot` | builds ChatGPT's search index — the one that feeds answers |
| `ChatGPT-User` | fetches your page when a person asks; robots.txt "may not apply" |
| `OAI-AdsBot` | validates a submitted ad page |

There is a second, larger hole, and it has nothing to do with which token you picked. One practitioner ran a study on this: 180 B2B SaaS and fintech domains, robots.txt checked against Wayback Machine timestamps, then 60 brand-name queries run through ChatGPT-4o between 15 June and 3 July. **31% of the domains blocking GPTBot were still cited** — with attribution, domain name and link. Most had been blocking for four months or more.

They traced where the citations came from:

- **About half — syndication.** Press-release networks, industry roundups, and content partners that had republished the original.
- **Roughly a third — forums and Q&A.** Reddit threads and Q&A sites where somebody had quoted or paraphrased the blocked domain.

That is one person's spreadsheet on one slice of the market, not a benchmark, and the exact percentage should not be treated as a law. The mechanism behind it is what generalises, and it is not really disputable:

> You can control whether a crawler hits your server. You can't control whether your content has already been copied, quoted, summarized, or cached somewhere the crawler CAN reach.
>
> — [r/GEO_optimization](https://www.reddit.com/r/GEO_optimization/comments/1v303a9/31_of_sites_blocking_gptbot_still_got_cited_by/)

robots.txt is a request-time control. It governs your server and nothing else. The moment a paragraph of yours exists on a page you do not own, your robots.txt has no jurisdiction over it — and, as that study points out, the citation credit then goes to whoever republished you.

One more thing worth knowing before you edit that file: the live fetchers sit outside robots.txt **by design**, and each vendor says so in its own documentation.

| Agent | Vendor | What the vendor says about robots.txt |
|---|---|---|
| `ChatGPT-User` | OpenAI | "robots.txt rules may not apply" |
| `Perplexity-User` | Perplexity | "generally ignores robots.txt rules" |
| `meta-externalfetcher` | Meta | "may bypass robots.txt rules" |

The reasoning is the same in all three cases — a person asked for that specific page, so it is treated as a user action rather than a crawl. And Amazon caches your robots.txt for up to 30 days, so a rule you add today may not take effect for weeks.

site: doableclaw.com/ai-visibility — See which AI engines are reading your site, and which ones are only claiming to.

## How do I check a bot is really who it says it is?

Match the requesting IP address against the ranges the vendor publishes. A user agent is a text field that anything can write, so the name in your log is a claim, not an identity — and the vendors know it. Common Crawl says outright that it is "aware of crawlers falsely identifying themselves as CCBot."

Every major operator publishes its address ranges as a JSON file for exactly this reason:

* OpenAI — a file per agent: `openai.com/gptbot.json`, `openai.com/searchbot.json`, `openai.com/chatgpt-user.json`, `openai.com/adsbot.json`
* Anthropic — one file for all its bots: `claude.com/crawling/bots.json`
* Perplexity — `perplexity.com/perplexitybot.json` and `perplexity.com/perplexity-user.json`
* Common Crawl — `index.commoncrawl.org/ccbot.json`

Google and Apple publish theirs alongside their crawler documentation, and Google additionally supports reverse DNS lookup as a check.

Two cautions before you build anything on those files:

- **A cached range list goes stale.** Each file carries a timestamp and changes, so ranges you saved months ago will produce false negatives on legitimate traffic.
- **IP blocking is not a substitute for robots.txt.** Anthropic notes that it "may not work correctly or persistently guarantee an opt-out" — true of every vendor whose infrastructure moves.

Verification is the step that turns a name into evidence, and it is where a hand-run log check starts getting genuinely laborious. Doing it once teaches you what the file can and cannot say. Doing it monthly, across four vendors and a moving list of ranges, is a different kind of job.

## Bottom Line

The single most useful thing you can do with a list of bot names is sort them into three buckets: training, index, and live. Everything else about your log reads differently once you have.

Two of the most-discussed names are not crawlers and will never appear. Blocking one agent from a vendor does not block its siblings, and blocking all of them does not stop your content being cited through somebody else's republished copy. The live fetchers, which are the most interesting rows you have, are largely outside robots.txt on purpose.

Take the names in your log as claims until you have checked them against the published ranges. Then remember what the whole file is and is not: good evidence that you are reachable, and no evidence at all about whether any answer named you.

## AI crawler user agents: frequently asked questions

### What is the difference between GPTBot and ChatGPT-User?

They fire at completely different moments. `GPTBot` collects training data on OpenAI's schedule, months before any model ships. `ChatGPT-User` fires when a real person asks something and ChatGPT fetches your page to help answer it. Neither is the one that builds ChatGPT's search index — that is `OAI-SearchBot`, a third agent with its own robots.txt token.

### Should I block AI crawlers in robots.txt?

That depends on whether you want the traffic or the training, and they are separable. Blocking the training crawlers while allowing the index crawlers keeps you eligible to be surfaced in AI search while opting out of model training. Blocking everything removes you from the index too, and — per the study above — still does not remove content of yours that lives on sites you do not control.

### Why do I see GPTBot but not Google-Extended?

Because Google-Extended does not make requests. It is a robots.txt control token that governs whether already-crawled content may train Gemini, and Google states it "doesn't have a separate HTTP request user agent string." `Applebot-Extended` works the same way. An empty result for either is normal for every site on the internet.

### Do AI crawlers respect robots.txt?

The training and index crawlers generally say they do. The live user-directed fetchers are the exception and say so plainly: OpenAI notes that for `ChatGPT-User`, "robots.txt rules may not apply"; Perplexity says `Perplexity-User` "generally ignores robots.txt rules"; Meta says `meta-externalfetcher` "may bypass robots.txt rules." The stated reasoning in each case is that a person requested that specific page.

### How do I verify a bot is not spoofed?

Check the requesting IP against the vendor's published range file rather than trusting the user agent string, which any script can forge. OpenAI, Anthropic, Perplexity and Common Crawl all publish JSON range lists; Google supports reverse DNS. Common Crawl explicitly warns about crawlers impersonating CCBot, so this is a real problem rather than a theoretical one.

### Is Amazonbot an AI crawler?

Partly. Amazon describes it as improving "our products and services," adding that the data "may be used to train Amazon AI models." It is not a chat engine crawler in the sense that GPTBot or ClaudeBot are, but it is frequently among the heaviest bots hitting a site, so it will distort any count that lumps all bots together.

### How often does this list change?

Often enough that a cached copy goes stale within months. Vendors add agents, split one into several, and rename them — Meta's AI search crawler, `meta-webindexer`, is a recent addition alongside the older `meta-externalagent`. Check the linked vendor pages rather than trusting any roster, including this one, past its verification date.

---

**Meta Title:** GPTBot, ClaudeBot and Every Other AI Crawler

**Meta Description:** GPTBot, ClaudeBot, OAI-SearchBot and the rest — what each is for, which ones fire when a person is actually asking, and the two that never appear in your log.