GPTBot, ClaudeBot and Every Other AI Crawler

You grepped your access log, and now you have a list of names. GPTBot. ClaudeBot. Meta-ExternalAgent. Something called OAI-SearchBot that you have never heard of.

They are not interchangeable, and the differences decide what you can do about any of them. Some are collecting training data. Some are building a search index. Some fired because a person was asking a question at that exact moment. Two of the names most commonly discussed are not crawlers at all and will never appear in your log.

Every row below is taken from the operator's own published documentation, linked, and checked on 2 September 2026. This roster changes — vendors add agents, rename them, and split one into three. If you are reading this much later, follow the links rather than the table.

Which AI Bots are These? the Full List

Here is every documented agent from the major vendors, grouped by who runs it. The column that matters most is when it fires, because that is what separates a bot indexing you in the background from a bot fetching your page because somebody asked about you thirty seconds ago.

Sources, in order: OpenAI, Anthropic, Perplexity, Google, Meta, Apple, Amazon, Common Crawl.

Names you will see that are not on this list are not necessarily fake — plenty of smaller engines and SEO tools crawl too, and some are documented nowhere useful. Treat an undocumented name as unknown rather than as an AI engine.

Why Every Vendor Now Runs Three Different Bots

Because three different things are happening, and lumping them together is what makes a log unreadable. Read down the table and the same shape repeats at OpenAI, Anthropic, Perplexity and Meta: a training crawler, a search-index crawler, and a live fetcher that runs when a user asks a question.

That middle row is the one worth caring about. Training crawls feed a model that ships months later. The index row is what a chat engine actually searches when it needs a current answer, so it is the closest thing in your log to being in the running for a citation.

The bottom row is closer still, and it is the most misread. A live fetch means somebody typed a question and the engine went and got your page to help answer it. It is the strongest single signal in the file. It is also not proof you were named in the answer it produced — a bot visiting is not the same as being mentioned, and the gap between the two is the thing that makes logs feel more informative than they are.

One practical consequence: counting "AI bot hits" as a single number destroys the only distinction that matters. A thousand training crawls and one live fetch is a very different week from the reverse, and a total tells you which one you had.

The Two Names You Will Never Find in Your Log

Google-Extended and Applebot-Extended are not crawlers. They are robots.txt control tokens — labels you use to grant or refuse permission — and neither one ever makes a request. Grep for them forever and you will find nothing, because there is nothing to find.

Both vendors say so themselves.

This trips people up in both directions. Blocking Google-Extended does not reduce your crawl traffic by a single request, because it was never making any. And an empty grep for these names is not evidence that Google or Apple are ignoring you — it is the expected result, always, for everybody.

The general rule: the permission layer and the request layer are different systems.

• The request layer — a bot fetches your page, and your server writes a line about it. This is the only one that leaves a trace.
• The permission layer — what may be done with your content after it has been collected. It never makes a request, so it never appears anywhere you can grep.

Does Blocking GPTBot Stop ChatGPT From Citing Me?

No — and OpenAI's own documentation is the fastest way to see why. GPTBot is described as the crawler for "content that may be used in training our generative AI foundation models." It is one of four OpenAI agents. OAI-SearchBot is the one "used to surface websites in search results in ChatGPT's search features," and it is governed by its own token.

So a robots.txt that disallows GPTBot and nothing else opts you out of training while leaving ChatGPT's search index entirely untouched. Many sites that believe they have blocked OpenAI have blocked exactly one of its four agents, and not the one that feeds answers.

Write Disallow: GPTBot and you have opted out of training data collection. Three OpenAI agents are still running:

There is a second, larger hole, and it has nothing to do with which token you picked. One practitioner ran a study on this: 180 B2B SaaS and fintech domains, robots.txt checked against Wayback Machine timestamps, then 60 brand-name queries run through ChatGPT-4o between 15 June and 3 July. 31% of the domains blocking GPTBot were still cited — with attribution, domain name and link. Most had been blocking for four months or more.

They traced where the citations came from:

• About half — syndication. Press-release networks, industry roundups, and content partners that had republished the original.
• Roughly a third — forums and Q&A. Reddit threads and Q&A sites where somebody had quoted or paraphrased the blocked domain.

That is one person's spreadsheet on one slice of the market, not a benchmark, and the exact percentage should not be treated as a law. The mechanism behind it is what generalises, and it is not really disputable:

robots.txt is a request-time control. It governs your server and nothing else. The moment a paragraph of yours exists on a page you do not own, your robots.txt has no jurisdiction over it — and, as that study points out, the citation credit then goes to whoever republished you.

One more thing worth knowing before you edit that file: the live fetchers sit outside robots.txt by design, and each vendor says so in its own documentation.

The reasoning is the same in all three cases — a person asked for that specific page, so it is treated as a user action rather than a crawl. And Amazon caches your robots.txt for up to 30 days, so a rule you add today may not take effect for weeks.

How Do I Check a Bot is Really Who It Says It Is?

Match the requesting IP address against the ranges the vendor publishes. A user agent is a text field that anything can write, so the name in your log is a claim, not an identity — and the vendors know it. Common Crawl says outright that it is "aware of crawlers falsely identifying themselves as CCBot."

Every major operator publishes its address ranges as a JSON file for exactly this reason:

• OpenAI — a file per agent: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json
• Anthropic — one file for all its bots: claude.com/crawling/bots.json
• Perplexity — perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json
• Common Crawl — index.commoncrawl.org/ccbot.json

Google and Apple publish theirs alongside their crawler documentation, and Google additionally supports reverse DNS lookup as a check.

Two cautions before you build anything on those files:

• A cached range list goes stale. Each file carries a timestamp and changes, so ranges you saved months ago will produce false negatives on legitimate traffic.
• IP blocking is not a substitute for robots.txt. Anthropic notes that it "may not work correctly or persistently guarantee an opt-out" — true of every vendor whose infrastructure moves.

Verification is the step that turns a name into evidence, and it is where a hand-run log check starts getting genuinely laborious. Doing it once teaches you what the file can and cannot say. Doing it monthly, across four vendors and a moving list of ranges, is a different kind of job.

Conclusion

The single most useful thing you can do with a list of bot names is sort them into three buckets: training, index, and live. Everything else about your log reads differently once you have.

Two of the most-discussed names are not crawlers and will never appear. Blocking one agent from a vendor does not block its siblings, and blocking all of them does not stop your content being cited through somebody else's republished copy. The live fetchers, which are the most interesting rows you have, are largely outside robots.txt on purpose.

Take the names in your log as claims until you have checked them against the published ranges. Then remember what the whole file is and is not: good evidence that you are reachable, and no evidence at all about whether any answer named you.