robots.txt and AI: the exhaustive list of bots to allow in 2026

You have the best content in your industry. You rank well on Google. And yet ChatGPT never mentions you. In half of these cases, it's not a content problem — it's your robots.txt or your CDN closing the door on AI bots without anyone noticing.

We hit this exact case on one of our own sites: livada.fr was returning 403s to GPTBot, ClaudeBot, and PerplexityBot because of a default CDN rule. Flawless site, closed door. Here's the exhaustive list of bots to allow in 2026, how to do it properly, and how to check that it works.

Why your site can be invisible to AI for no apparent reason

Three possible sources of blocking, often stacked together:

• Your robots.txt contains a Disallow: / targeted at specific user-agents — sometimes set years ago by an "anti-scraping" SEO plugin.
• Your CDN or host (Cloudflare, Hostinger, OVH, certain managed hosts) blocks AI bots by default at the firewall level, before your robots.txt is even read.
• A WAF (Web Application Firewall) triggers a challenge on non-browser user-agents.

Result: the AI gets a 403 or a JavaScript challenge and gives up. Your site exists for Google, not for AI.

The two families of AI bots — and why the distinction matters

AI bots fall into two distinct families. Blocking them together is a common mistake.

1. Live citation bots (must be allowed)

These bots come to read your site the moment a user asks the AI a question. If you block them, you will never be cited in the answer — no matter how good your content is.

• OAI-SearchBot — OpenAI's search engine (ChatGPT Search).
• ChatGPT-User — triggered when a ChatGPT user clicks "search the web."
• Claude-User — triggered when a Claude.ai user turns on web search.
• Claude-SearchBot — Anthropic's dedicated search component.
• PerplexityBot — Perplexity's engine (real-time citations).
• Perplexity-User — fetch triggered by a Perplexity user.

2. Training / grounding bots (depending on your strategy)

These bots collect content to train or enrich the models. Allowing them increases your chances that your brand becomes known to AI as an entity. Blocking them protects your paid or proprietary content. This is a trade-off to weigh based on your business.

• GPTBot — OpenAI's training crawler.
• ClaudeBot — Anthropic's training crawler.
• Google-Extended — Google's signal for Gemini/AI Overviews (not to be confused with Googlebot — blocking Google-Extended doesn't affect classic Google SEO).
• Applebot-Extended — Apple's equivalent for Apple Intelligence.
• CCBot — Common Crawl (used by many open-source models).
• Bytespider, Amazonbot, Meta-ExternalAgent — the ByteDance/Amazon/Meta equivalents.

A clean robots.txt, ready to paste

Here's a balanced template: open to live citation bots (essential to get cited), open to the main training bots (to be recognized as an entity), and blocking commercial SEO scrapers that eat up bandwidth for nothing in return.

User-agent: *
Allow: /

# === AI bots — live citation (allow) ===
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /

# === AI bots — training / grounding ===
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: CCBot
Allow: /

# === Commercial SEO scrapers (optional) ===
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml

The silent 403 error — check your CDN

Having a permissive robots.txt isn't enough. If your CDN blocks the user-agent at the firewall level, the bot will never even read your robots.txt — it will get a 403 and move on.

Cloudflare

Cloudflare dashboard → your domain → Overview → the "Control AI crawlers" panel. Set "Block AI training bots" to Do not block. And switch "Manage your robots.txt" to Disable robots.txt configuration so your own file stays the only source.

Hostinger, OVH, and shared hosting

Check the WAF rules and the blocked user-agent lists in the control panel. Explicitly ask support to unblock GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot.

How to check that it's working (in 30 seconds)

Three ways, from the fastest to the most thorough:

1. Command-line test (instant)

From a terminal:

curl -A "GPTBot" -I https://yoursite.com/
curl -A "ClaudeBot" -I https://yoursite.com/
curl -A "PerplexityBot" -I https://yoursite.com/

You want to see an HTTP/2 200. If you get a 403, a 429, or an HTML challenge, the bot is blocked.

2. Free tool — Livada Radar

One of the signals measured by Livada Radar is precisely AI bot access to your site. Enter your URL, and the tool tells you plainly which bots can get in and which are blocked. Free, 30 seconds.

→ Test AI bot access on my site

3. Full audit with Livada SEO

Livada SEO publishes a clean robots.txt on WordPress right from installation, checks bot access continuously, and triggers a warning if a new block appears on the CDN side.

Once the door is open, what changes?

Opening access to AI bots is a necessary but not sufficient condition. To go from "visible" to "cited," you also need:

• Clean structured data (LocalBusiness, Hotel, Campground, FAQ schema).
• An anchored entity identity (Wikidata, sameAs, llms.txt).
• Content that answers your customers' real questions (not generic service pages).

But without the door open, none of this has any effect. Start there.

Be found by Google. And by AI.

"Found by Google. Cited by AI." In 2026, a site invisible to AI bots is a site missing out on half of all purchase intent. The diagnosis takes 30 seconds — test your site.