Which AI crawlers should you let in?

Three jobs, and a name for each

An AI company's crawler reads your page for one of three reasons, and the companies below give each job its own name, so robots.txt can answer each one separately.

Building an answer index. These read so that an assistant can find your page and cite it when somebody asks. Refusing one is how a site leaves that assistant's answers.

Fetching a page because a person asked. These open one page when somebody asks an assistant about it or pastes your link, the closest thing to a visit without a visitor.

Collecting pages to train models. These read to train future models, and refusing them is a decision about training alone.

Perplexity says neither of its crawlers collects pages for training, and DuckDuckGo says the same of its own.

Names that are not crawlers

Two names are written in robots.txt without a crawler of their own behind them.

• Google-Extended decides whether pages Google's ordinary crawlers have already read may train Gemini models or ground its answers. Google says it has no effect on inclusion or ranking in Google Search.
• Applebot-Extended decides whether pages Applebot has read may train Apple's foundation models. Apple says it crawls nothing itself, and a page refusing it can still appear in Apple's search results.

CCBot is a crawler, but of a different kind: it builds Common Crawl, an open archive of the web that anybody can download. What it is used for after that is up to whoever downloads it.

What robots.txt can and cannot do

robots.txt is a request, not a lock. The standard itself says its rules are not a form of access authorization, and three of these companies say a fetch made because a person asked may not follow it: OpenAI of ChatGPT-User, Perplexity of Perplexity-User, and Meta of Meta-ExternalFetcher. A page that must stay private needs a sign-in, not a line in robots.txt.

Which to let in

1. Let the answer crawlers in. If OAI-SearchBot, Claude-SearchBot or PerplexityBot cannot read a page, those assistants cannot cite it.
2. Leave the fetchers for people alone. Each one is somebody asking about your page right now, and refusing it may not stop it anyway.
3. Decide about training on its own. Refusing GPTBot, ClaudeBot or Google-Extended is a choice about future models, and on its own it does not take you out of answers or out of search.

A robots.txt that keeps a site in answers and says no to training:

# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everybody else, answer engines included
User-agent: *
Allow: /

Sitemap: https://salon.example/sitemap.xml

Leave the first group out and training crawlers are welcome too. Either way, nothing in it refuses the crawlers that let an assistant cite you.

Seeing who actually came

BackendFree recognises 23 AI crawlers by name, counts each one apart from people, and none of them spends the events allowance. The crawler panel on the traffic screen says which came and how often, so you can see whether a refusal is being honoured.

Most crawlers never run JavaScript, so a script in the page cannot see them: they are counted through the fallback image in the snippet, or a report from your own server, and the figures are a floor. Is ChatGPT sending you visitors? covers the other half: the people an assistant sends. Web analytics has the rest of what the measurement does.