How to set up robots.txt for AI crawlers without locking yourself out of AI answers

Your robots.txt decides whether AI engines are allowed to fetch your pages. Get it wrong in one direction and you are training other people's models for free with no visibility in return. Get it wrong in the other and ChatGPT, Perplexity and Claude cannot cite you at all. Most sites we audit have not made a decision either way; they have inherited a file from a theme, a plugin or a block list someone pasted in 2024.

This guide gives you a decision framework, the agent names to use (spelled exactly as the vendors publish them), and a template you can adapt.

Step 1: Understand the three jobs an AI bot can have

Every major AI company runs more than one crawler, and they do different things.

Blocking a training agent is a business decision about your content being used to build models. Blocking a search or user agent is a visibility decision: you will not appear in that engine's answers. Treat them separately.

Step 2: Decide your policy

Three sensible positions:

1. Open: allow everything. Simplest, maximum visibility, your content may be used for training. Right for most small businesses whose content is marketing rather than a product.
2. Visible but not training: allow all search and user agents, disallow the training agents. Right for publishers, consultancies and anyone whose content is the product.
3. Closed: block all AI agents. Only right if you have decided AI visibility does not matter to you. Be honest about whether that is a decision or a default.

Whatever you pick, Googlebot and Bingbot must stay allowed. They power the web index behind Google's AI features and ChatGPT search respectively.

Step 3: Write the rules

The template below is position 2. For position 1, delete the Disallow: / lines under the training agents. Each User-agent group needs at least one Allow or Disallow line or it does nothing.

# --- Search and retrieval crawlers: keep these allowed to appear in AI answers ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Applebot
Allow: /

User-agent: Amazonbot
Allow: /

# --- Training crawlers: this section opts out of model training only ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# --- Everyone else, including Googlebot and Bingbot ---
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Two details matter. Spelling must match the vendor's token exactly (GPTBot, not GPT-Bot). And a bot only uses the most specific group that names it; if you have a User-agent: * group that disallows a folder, a named agent with its own group does not inherit that rule, so repeat any path exclusions you care about inside the named groups.

Step 4: Make sure the firewall agrees

Robots.txt is read after the request reaches your server. If Cloudflare, your host or a security plugin returns a 403 first, the file is irrelevant. After editing robots.txt, check the bot settings on your CDN and any WordPress security plugin, and switch off blanket "block AI bots" toggles if your policy is to be visible. Then confirm in your logs that OAI-SearchBot, PerplexityBot and Claude-SearchBot get 200s.

Step 5: Test it

• Google Search Console has a robots.txt report under Settings that shows the fetched file and any parse errors.
• Bing Webmaster Tools has a robots.txt tester.
• For AI agents, fetch the file yourself and check the groups are separated by blank lines with no stray characters. Then run the curl test from our guide on checking whether AI engines can read your site, using the OAI-SearchBot user agent, and confirm a 200.

Step 6: Review it twice a year

Vendors add agents. OpenAI split GPTBot and OAI-SearchBot; Anthropic added Claude-SearchBot; Perplexity added Perplexity-User. Put a calendar reminder in for January and July to check the vendor docs linked below and update the file.

Common mistakes

• Pasting a "block all AI" list from a forum in 2024 that includes OAI-SearchBot and PerplexityBot, then wondering why the business never appears in ChatGPT.
• Blocking Google-Extended believing it protects rankings. It has no effect on Search; it only limits Gemini training use.
• A User-agent: * group with Disallow: / left over from a staging site. Every AI bot without a named group obeys it.
• Assuming robots.txt stops bad actors. It does not. Use rate limiting and WAF rules for that.