You probably blocked the AI crawlers. Most sites did it by accident.

In 2024 the conversation about AI crawlers was about protecting content from being used to train models without permission. It was a reasonable concern and a lot of people acted on it: they added block lists to robots.txt, they turned on the new "block AI bots" toggles in Cloudflare and in WordPress security plugins, and they moved on.

Two years later the same crawlers, or their siblings, decide whether a business appears in ChatGPT, Perplexity and Claude when a customer asks for a recommendation. And the blocks are still there. In our audits, accidental blocking is the most common single reason a site with decent content is completely absent from AI answers. Nobody chose it. It is a default that outlived the reason for it.

Three ways it happens

The CDN did it. Cloudflare began blocking AI crawlers by default on new domains in mid-2025 and has kept adjusting the settings since. Other CDNs and hosts added similar managed rules. The result is a 403 at the edge that your robots.txt never gets a chance to override. The site looks open in every file you control and is closed in a dashboard nobody has visited.

A plugin did it. Wordfence, Sucuri, All In One Security and a dozen others shipped bot-blocking features with AI crawlers pre-listed. One checkbox, ticked once during a security review, and OAI-SearchBot has been getting a 403 ever since.

A block list did it. Someone pasted a list from a blog post into robots.txt. The list was written when GPTBot was the only OpenAI agent, so it blocks "GPTBot" and, in later versions, everything else OpenAI, Anthropic and Perplexity run. The person meant "do not train on us". The file says "do not cite us".

The distinction nobody explained

Each major AI company runs separate crawlers for separate jobs. OpenAI has GPTBot for training, OAI-SearchBot for the search index behind ChatGPT search, and ChatGPT-User for fetching a page a user has asked about. Anthropic mirrors that with ClaudeBot, Claude-SearchBot and Claude-User. Perplexity runs PerplexityBot for its index and Perplexity-User for live fetches, and says neither is used for training. Google-Extended is not even a crawler; it is a token that controls whether content fetched by Googlebot can be used for Gemini training.

Blocking a training agent is a legitimate decision about your intellectual property. Blocking a search or user agent is a decision to be absent from that engine's answers. They are different decisions, and the 2024 block lists collapsed them into one.

How to tell if it happened to you

Ten minutes:

1. Open your robots.txt. Look for OAI-SearchBot, PerplexityBot, Claude-SearchBot and a catch-all User-agent: * with Disallow: /.
2. Log in to your CDN and look for anything with "AI" or "bot" in the name under Security.
3. Open your WordPress security plugin and look for a bot-blocking section.
4. Fetch a page pretending to be a search crawler: curl -I -A "OAI-SearchBot" https://yourdomain.com/ and check for a 200, not a 403.
5. Search your server logs for the last month for OAI-SearchBot and PerplexityBot. No hits, or only 403s, tells you what you need to know.

Make the decision on purpose

You may still decide to block training crawlers. Publishers and businesses whose content is their product often do, and OpenAI, Anthropic and Google all document how to do that without touching search visibility. What we are asking is that it be a decision. Write down your policy, set robots.txt and the CDN to match it, and check both twice a year, because the vendors keep adding agents.

The businesses winning in AI search right now are not doing anything clever. Many of them are simply the ones that were not blocked while their competitors were.