Should you let AI crawlers read your site

Over the past year, more and more people have noticed unfamiliar names in their traffic logs: GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended. These are not people. They are programs sent by AI companies to read your website. Which raises a question: should you shut them out?

The advice online splits hard. One camp says “they are stealing your content to train their models — block all of them now.” The other says “block them and you vanish from the answers AI gives, which is cutting off your own customers.” Both are right, and both are incomplete — because they are not talking about the same kind of crawler.

This piece is an attempt to make the whole thing clear: what kinds of crawlers there are, what actually happens when you block each, how a business like yours should decide, and then how to configure it and — critically — how to check that your configuration does what you think. It runs long, because getting this decision wrong means either quietly losing traffic that should have been yours, or failing to protect the very thing you set out to protect.

First, separate the two kinds of crawler — everything hinges on this

Treating “AI crawlers” as one thing is the source of nearly every mistake. They are at least two different crowds, with opposite purposes.

The first kind: training crawlers. They fetch your pages to feed the next generation of models as training material. Your content is digested into the model’s parameters, and later, when someone asks a related question, the model may answer from memory — but usually without telling the user the line came from you, and without a link. The headline examples are OpenAI’s GPTBot, the non-profit CCBot (Common Crawl, which almost every large model has trained on), and the two “training switches” Google-Extended and Applebot-Extended (more on why those are special below).

The second kind: retrieval crawlers, also called live crawlers. These arrive at the exact moment someone is asking a question, fetch your page, and immediately use it to assemble an answer — and they usually cite the source and link to it. The headline examples are OpenAI’s OAI-SearchBot and ChatGPT-User (used when ChatGPT goes online), PerplexityBot, and Anthropic’s Claude-User for live fetches.

See the difference —

• Block the training crawlers and what you lose is the chance to be written into the model’s memory. That loss is slow, indirect, and genuinely contested — some people simply do not want to be training fodder.
• Block the retrieval crawlers and what you lose is the chance to be read at the very moment someone is asking about your industry. That loss is immediate, direct, and is often the exact customer who should have clicked through.

The overwhelming majority of people who say “I blocked all the AI crawlers” meant to stop the first kind, but with a blanket rule caught the second as well. The result: they did not actually keep their content out of training (more on that below), yet they very much did disappear from AI’s live answers. This is the most common — and most costly — misconfiguration there is.

There is a third category people miss: search-index crawlers. Google’s own Googlebot, for instance, fetches pages to build the search index — and Google’s AI Overviews (the AI summary at the top of the results page) draw on that same index. So blocking Googlebot to “keep AI out” also switches off ordinary Google search. This is exactly why Google invented the token Google-Extended: it is not a crawler at all, it is a signal — put it in your robots.txt and Googlebot still crawls for Search, but Google undertakes not to use your content to train Gemini. Apple’s Applebot-Extended works the same way.

What they are actually called: a reference list

Each vendor documents its crawlers on its own pages, and the list changes every few months. Below are the main players at the time of writing, grouped by what blocking them costs you — to help you decide, not to memorise.

Training-side (blocking = kept out of the model’s memory; no effect on your visibility in search or live answers):

• GPTBot — OpenAI’s training fetch.
• CCBot — Common Crawl, a public dataset used indirectly by countless models. Blocking it has the widest reach.
• Google-Extended — not a crawler; the switch for “may Google use this to train Gemini.”
• Applebot-Extended — the same, for Apple’s AI.
• Meta-ExternalAgent — the training fetch for Meta (parent of Facebook and Instagram).
• Bytespider — ByteDance (parent of TikTok). Poor reputation; covered separately below.
• Amazonbot, anthropic-ai and others — various vendors also run training- or research-oriented fetches.

Retrieval / live-side (blocking = you vanish from that product’s live AI answers):

• OAI-SearchBot — builds the index behind ChatGPT’s search.
• ChatGPT-User — the live fetch when a user tells ChatGPT to “go look that up.”
• PerplexityBot / Perplexity-User — Perplexity’s index and live fetch.
• Claude-User / Claude-SearchBot — Claude’s fetches when it goes online or searches.
• DuckAssistBot — DuckDuckGo’s assistant.

The pattern is clear: names containing Search or User are usually retrieval-side, and blocking them costs you visibility directly; the purely training ones are a separate calculation. But do not take that as gospel — naming is inconsistent, and one company’s crawler can serve both purposes (Anthropic’s ClaudeBot, for one, has historically been mainly a training fetch). If you need precision, read each vendor’s own crawler documentation, and trust no list written from memory — including this one.

So how should a business like yours decide

Set the ideology aside and the decision comes down to one thing: is your content itself the thing you sell?

If you are an ordinary business that turns content into customers

Dentists, builders, immigration advisers, accountants, restaurants, tutoring centres — most New Zealand small businesses are here. Your website is a customer-acquisition tool, not a paid product. Someone finds you, understands you, then gets in touch; that is the whole value of the site.

For you, the answer is almost certainly: allow every retrieval crawler, and there is little reason to block the training ones either. The reasoning —

• Retrieval crawlers bring you the person who is right now looking for what you do. Someone asks ChatGPT “who builds bilingual websites in Auckland” — you want it to be able to read you, name you, link you. Blocking it hands that customer to a competitor. There is no trade-off to weigh here.
• Training crawlers look like they “take your content for free,” but for an acquisition-driven business, being written into the model’s memory is a good thing. It means that later, when someone offline asks “roughly what does a business website cost in New Zealand,” the model may answer, from what it learned, with information tied to you. The more your brand name, price range and scope of service recur in the training data, the more likely you are to be mentioned in exactly these sourceless answers.
• The content you would be “protecting” — the services on your homepage, a handful of blog posts — is content you want seen by as many people as possible. Being summarised by an AI widens your reach. You have no exclusive material you need to hide.

So for this kind of business our advice is almost boringly plain: block nothing, and spend the effort on making your content worth quoting instead. How to make content worth quoting is another article — that is exactly what GEO is about.

If your content is your product

News outlets, paid research, original photography, large libraries of original tutorials, recipe sites, in-depth review sites — these are different. If a reader “getting what they need without clicking through or contacting you” leaves you worse off, that is because your business model monetises the content itself (subscriptions, ads, licensing).

For these sites the worry is real: training crawlers ingest years of original work into a model for free, the model then generates “something close enough,” and you get neither a cent nor a click. This is why several large publishers spent last year blocking GPTBot and, in some cases, going to court.

But even here the mature move is not a blanket block — it is to split the two:

• Consider blocking the training crawlers (GPTBot, CCBot, Google-Extended, Applebot-Extended and so on) so your core asset does not enter models for free.
• Keep the retrieval crawlers allowed — because when AI answers live, it links out and sends traffic. That traffic is the point. Blocking PerplexityBot and OAI-SearchBot too is turning away readers arriving at your door.

This “block training, allow retrieval” combination is the careful work worth doing for a site whose content has genuine standalone value. Most small businesses are not in this category — and if you are wondering whether you count, you almost certainly do not.

A hard truth: robots.txt cannot stop anyone determined to crawl

Suppose you have decided to block. A splash of cold water first: robots.txt is a “please do not enter” note, not a wall.

It works by gentlemen’s agreement — a crawler voluntarily reads your robots.txt on arrival and, seeing a Disallow, voluntarily declines to fetch. The well-behaved majors (OpenAI, Google, Anthropic) do honour it; that is their public commitment, and breaking it carries reputational and legal risk.

Others are less well-behaved. Bytespider has been reported repeatedly by security firms and site owners to ignore robots.txt and crawl anyway, aggressively. Against a crawler like that, a hundred lines of Disallow do nothing — it does not read the file, or reads it and shrugs.

If you want to actually block rather than politely request, you need something harder:

• Block at the edge, by user agent or IP. Cloudflare, for example, now has a one-click “block AI bots” switch (under Bot management / AI Audit) that stops known AI crawlers by signature before the request reaches your server, without relying on the crawler’s own conscience. That is blocking in the real sense.
• Server-level rules (Apache/Nginx returning 403 by user agent) work on the same idea, but a user agent can be faked, so it is weaker than edge-side identification.

So the complete mental model is: robots.txt expresses your wishes to the well-behaved majors (and does work on them); edge blocking like Cloudflare’s deals with the ones that do not behave. Writing only the former and assuming you are covered is a common bit of self-reassurance.

Another overlooked cost: server load

Even if you do not care whether your content is used for training, there is a more practical reason you might want to rein crawlers in: they consume your server’s resources.

Some AI crawlers fetch very intensively — hundreds or thousands of requests in a short window, sweeping the whole site top to bottom, then coming back a few days later. The impact varies enormously from site to site:

• A static site behind a CDN (say, one hosted on Cloudflare Pages) is barely affected. The pages are pre-built plain files, served to the crawler at the edge, never troubling your origin. Ten thousand fetches cost you roughly nothing. This is one of the side benefits of why we default clients to static hosting.
• A dynamic site (WordPress, anything with a database) feels it. Every page a crawler fetches means your server runs PHP and hits the database live. Intensive crawling is indistinguishable from a small spike of real traffic — and on cheap shared hosting it can even slow down genuine visitors.

If you are in the latter situation and find one crawler hammering you in the logs, rate-limiting or blocking it is a pure operations decision, unrelated to “do I want to be used by AI.” In that case block the specific, aggressive user agent — not every AI indiscriminately.

Copyright and “zero-click”: worth thinking through, not worth panicking over

The core feeling behind opposing AI crawlers is really this: I worked hard on this, an AI sums it up in one sentence, and the user leaves satisfied without ever visiting my site. The phenomenon has a name — “zero-click”: the answer is delivered at the AI, and you never even get the visit.

The worry is real, but it matters to different businesses differently:

• If you monetise page views (ads, subscriptions), zero-click hits your revenue directly and deserves to be taken seriously.
• If you monetise conversions (someone contacts you, orders, books), it is the opposite — the AI has already explained “what this firm does, roughly what it costs, whether it seems trustworthy,” so the person who then comes to you has pre-qualified themselves, and is closer to buying. Here being summarised is not a loss; it is free first-round screening.

In other words, the same event is a threat to a publisher and an assist to most service businesses. Work out what you actually earn money from before deciding whether zero-click is your enemy — do not panic along with the crowd.

How to configure it

Here are three tiers, from “do nothing” to “block what can be blocked.” robots.txt lives at the root of your site (yourdomain/robots.txt) and is plain text.

Tier one: allow everything (recommended for the vast majority of small businesses). There is nothing to do — no Disallow means allowed by default. The positive action worth taking is to actively help AI understand you:

• make sure key content is in the HTML, not painted in by script (most crawlers do not run JavaScript);
• express your company details as structured data;
• publish an llms.txt — a plain text file at the site root that states, as concisely as possible, what you do, your main pages, your services and prices. It is the positive counterpart to robots.txt: robots.txt says “do not fetch this,” llms.txt says “here is what matters, please read it.” Ours is public — open it as a reference, there is nothing secret about the format.

Tier two: refuse training, keep search and live answers (recommended for content-monetised sites). Block the training side only, and leave retrieval and search alone:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

Note this does not block OAI-SearchBot, ChatGPT-User, PerplexityBot or Googlebot — so you still appear in ChatGPT’s online search, in Perplexity, and in ordinary Google, you simply stay out of each vendor’s training corpus. This is the “block training, allow retrieval” split from earlier.

Tier three: block AI as far as you can. On top of tier two, add the retrieval side (OAI-SearchBot, ChatGPT-User, PerplexityBot / Perplexity-User, Claude-User / Claude-SearchBot, DuckAssistBot and so on), each with its own Disallow: /. But keep two things in mind: one, this makes you disappear from AI’s live answers, which for most businesses is self-harm; two, it only works on well-behaved crawlers — to actually stop something like Bytespider you still need the Cloudflare edge blocking above.

One more trick: if you only want to block specific pages (some deep piece you would rather not have summarised) rather than the whole site, use Disallow: /that-path/ to target a directory and leave the rest open. robots.txt can be as fine-grained as you like.

Once configured, always verify — do not trust “I changed it so it works”

robots.txt is the easiest thing to “think you set up when you did not” — a wrong path, a syntax slip, or a cache in the way will all leave your rules toothless. Spend two minutes checking:

• Confirm the file is really there and really says what you think. Run curl https://yourdomain/robots.txt and see whether it returns what you wrote. A browser works too, but curl is cleaner and not fooled by browser caching. (“I changed it but nothing happened” is usually caching.)
• See who has actually visited. If you can get at your server access logs, filtering by user agent tells you which AI crawlers really came and how often. That beats any guess — you may find the crawler you fret about never showed up, while one you had never heard of crawls hardest.
• Re-test visibility the way a person would. The ultimate point of the config is whether AI mentions you. So periodically ask ChatGPT, Claude and Perplexity the way a prospective customer would, and watch whether the result shifts. We wrote a separate piece on running that self-check.

Our position, in one line

For the vast majority of New Zealand small businesses: do not block. Treat AI crawlers as a free exposure channel, then put the effort into making your content worth quoting. You are far more likely to lose out because AI cannot read you than because it can.

Only when your content is itself the product you sell is “block training, allow retrieval” worth configuring carefully — and either way, if what you need is to genuinely stop crawlers rather than politely ask, robots.txt is not enough on its own; pair it with edge blocking.

The last line is the most important, and the real point of this whole piece: do not, in order to “protect” content you would in fact love more people to see, shut off the very channel that brings customers to your door. It is the mistake we see most often, and the most regrettable.

If you are not sure what stance your site currently takes toward AI crawlers, or you want your visibility tested systematically, our SEO and GEO service covers exactly this. Though you are entirely able to walk through the method above yourself — it is all written out.