# Which AI crawlers should you let in?

> Some read to train a model, some to build an answer index, and some fetch a page because a person asked. Refusing the first kind keeps you in the answers.

- Let the answer crawlers in, or assistants cannot cite you.
- Decide about training on its own: refusing it does not hide you from answers.
- 23 AI crawlers are named one by one, and none spends the events allowance.

## Three jobs, and a name for each

An AI company's crawler reads your page for one of three reasons, and the
companies below give each job its own name, so robots.txt can answer each one
separately.

**Building an answer index.** These read so that an assistant can find your
page and cite it when somebody asks. Refusing one is how a site leaves that
assistant's answers.

| Company | Name |
|---|---|
| OpenAI | `OAI-SearchBot` |
| Anthropic | `Claude-SearchBot` |
| Perplexity | `PerplexityBot` |
| DuckDuckGo | `DuckAssistBot` |

**Fetching a page because a person asked.** These open one page when somebody
asks an assistant about it or pastes your link, the closest thing to a visit
without a visitor.

| Company | Name |
|---|---|
| OpenAI | `ChatGPT-User` |
| Anthropic | `Claude-User` |
| Perplexity | `Perplexity-User` |
| Mistral AI | `MistralAI-User` |
| Meta | `Meta-ExternalFetcher` |

**Collecting pages to train models.** These read to train future models, and
refusing them is a decision about training alone.

| Company | Name |
|---|---|
| OpenAI | `GPTBot` |
| Anthropic | `ClaudeBot` |
| Meta | `meta-externalagent` |

Perplexity says neither of its crawlers collects pages for training, and
DuckDuckGo says the same of its own.

## Names that are not crawlers

Two names are written in robots.txt without a crawler of their own behind them.

- **`Google-Extended`** decides whether pages Google's ordinary crawlers have
  already read may train Gemini models or ground its answers. Google says it has
  no effect on inclusion or ranking in Google Search.
- **`Applebot-Extended`** decides whether pages Applebot has read may train
  Apple's foundation models. Apple says it crawls nothing itself, and a page
  refusing it can still appear in Apple's search results.

`CCBot` is a crawler, but of a different kind: it builds Common Crawl, an open
archive of the web that anybody can download. What it is used for after that is
up to whoever downloads it.

## What robots.txt can and cannot do

robots.txt is a request, not a lock. The standard itself says its rules are not
a form of access authorization, and three of these companies say a fetch made
because a person asked may not follow it: OpenAI of `ChatGPT-User`, Perplexity
of `Perplexity-User`, and Meta of `Meta-ExternalFetcher`. A page that must stay
private needs a sign-in, not a line in robots.txt.

## Which to let in

1. **Let the answer crawlers in.** If `OAI-SearchBot`, `Claude-SearchBot` or
   `PerplexityBot` cannot read a page, those assistants cannot cite it.
2. **Leave the fetchers for people alone.** Each one is somebody asking about
   your page right now, and refusing it may not stop it anyway.
3. **Decide about training on its own.** Refusing `GPTBot`, `ClaudeBot` or
   `Google-Extended` is a choice about future models, and on its own it does not
   take you out of answers or out of search.

A robots.txt that keeps a site in answers and says no to training:

```text title=robots.txt
# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everybody else, answer engines included
User-agent: *
Allow: /

Sitemap: https://salon.example/sitemap.xml
```

Leave the first group out and training crawlers are welcome too. Either way,
nothing in it refuses the crawlers that let an assistant cite you.

## Seeing who actually came

BackendFree recognises 23 AI
crawlers by name, counts each one apart from people, and none of them spends
the events allowance. The crawler panel on the traffic screen says which came
and how often, so you can see whether a refusal is being honoured.

Most crawlers never run JavaScript, so a script in the page cannot see them:
they are counted through the fallback image in the snippet, or a report from
your own server, and the figures are a floor. [Is ChatGPT sending you
visitors?](/guides/is-chatgpt-sending-you-visitors/) covers the other
half: the people an assistant sends. [Web analytics](/product/web-analytics/)
has the rest of what the measurement does.

## Questions people ask next

### Does blocking GPTBot take my site out of ChatGPT?

**No.** OpenAI reads for ChatGPT's search answers with OAI-SearchBot and for training with GPTBot, and robots.txt names the two apart. Refusing GPTBot leaves you in the answers.

### Does refusing Google-Extended affect Google Search?

**No.** Google says it has no effect on inclusion or ranking in Search. It decides whether pages Google's crawlers already read may train Gemini models or ground its answers.

### Do fetchers acting for a person follow robots.txt?

**Not always.** OpenAI, Perplexity and Meta each say a fetch made because a person asked may not follow robots.txt. Treat those as a visitor arriving through an assistant.

### Is robots.txt enough to keep a page private?

**No.** The standard itself says its rules are not a form of access authorization. A page that must stay private needs a sign-in, not a line in robots.txt.

## Sources

- [OpenAI crawlers](https://developers.openai.com/api/docs/bots), OpenAI
- [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), Perplexity
- [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), Google Search Central
- [About Applebot](https://support.apple.com/en-us/119829), Apple
- [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers), Meta
- [Is DuckAssistBot related to DuckDuckGo?](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), DuckDuckGo
- [Mistral crawlers](https://docs.mistral.ai/robots), Mistral AI
- [CCBot](https://commoncrawl.org/ccbot), Common Crawl
- [RFC 9309, the Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), IETF