AI Bot Access
v1.12+ LIVADA EXCLUSIVEControl directly from WordPress which AI crawlers can read your site. Live HTTP diagnostics per bot, strict server-side blocking, enriched robots.txt — the "And by AI" promise made measurable and configurable.
1 Why this feature exists
ChatGPT, Claude, Perplexity, Gemini, Copilot — when a user asks them a question, these assistants crawl dozens of sites live to build their answer and cite sources. But most WordPress sites block these bots by default (inherited Cloudflare configuration, host WAF, managed robots.txt) — without the site owner knowing.
Your site can be 100/100 SEO, your content can be perfect — if Cloudflare or your host blocks GPTBot/ClaudeBot, you will never appear in AI answers. And you won't know it.
A live diagnostic tells you in 10 seconds exactly which bot gets through (200) or is blocked (403). You see where to act, and you control access directly from WordPress.
No other WordPress SEO plugin (Yoast, Rank Math, SEOPress, AIOSEO) exposes this control. The "AI Bot Access" feature is exclusive to Livada SEO.
2 How it works
The feature combines 3 complementary mechanisms:
- 1 Enriched robots.txt
The plugin rewrites your
robots.txtwith an explicitUser-agent+Allow: /orDisallow: /block per bot. No ambiguity for well-behaved crawlers. - 2 Strict blocking (real HTTP 403)
For scrapers that ignore
robots.txt(competing SEO scrapers, opaque scrapers), the plugin returns a real HTTP 403 from PHP at theparse_requestlevel — well before the theme or any DB queries load. Negligible performance impact (one user-agent string comparison per request). - 3 Live diagnostics (curl per bot)
Clicking "Run live diagnostics" sends a real HTTP request to your site using each bot's exact user-agent and displays the returned HTTP code (colored green / yellow / red). If a bot you've allowed in the plugin returns 403, the block is coming from the edge (Cloudflare AI Crawl Control, host WAF) — not WordPress. The diagnostic tells you where to look.
3 Live diagnostics — the demo that convinces in 30 seconds
On the Livada SEO → AI Bot Access admin page, the "Run live diagnostics (test all bots)" button runs, in ~10-30 seconds, one HTTP request per bot using its real user-agent. For each bot, you see:
Typical use case: you've enabled all AI bots in the plugin, the diagnostic shows Googlebot at 200, but ClaudeBot / PerplexityBot / GPTBot all at 403. Immediate diagnosis: your CDN (Cloudflare or host) is blocking AI. Head to your CDN's "AI Crawl Control" panel to allow it.
4 Strict blocking mode (Hard block)
robots.txt is a directive:
well-behaved bots (Googlebot, GPTBot, Claude) obey it. But competing SEO scrapers
(SemrushBot, AhrefsBot, Majestic MJ12bot, Moz DotBot, DataForSeoBot,
BLEXBot, PetalBot Huawei) deliberately ignore robots.txt and keep
draining your bandwidth without sending you a single visitor.
Strict blocking mode (enabled by default) returns a real HTTP 403 from PHP to these scrapers. The request is stopped at the server level, before any theme loads or database query runs.
| Behavior | robots.txt alone | Hard block |
|---|---|---|
| Well-behaved bot (Googlebot, GPTBot) | Obeys ✓ | Obeys or 403 ✓ |
| SEO scraper (SemrushBot, AhrefsBot) | Ignores it, keeps going ✗ | HTTP 403 ✓ |
| Performance cost per request | 0 ms | ~0.1 ms (string match) |
5 The 33 bots covered, sorted into 6 categories
Search engines
Googlebot · Bingbot · Applebot · DuckDuckBot
Classic crawlers. Never block these except in specific cases.
AI citation
OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · ClaudeBot · Claude-SearchBot · Claude-User · MistralAI-User · DuckAssistBot · Meta-ExternalFetcher
AI assistants that crawl on demand to cite your site. This is the core of "And by AI" visibility.
AI training
GPTBot · CCBot · Google-Extended · anthropic-ai · Amazonbot · Bytespider · FacebookBot · Meta-ExternalAgent · Google-CloudVertexBot · cohere-ai
Feed model training corpora. Allowing = your site becomes "known" to future models. Blocking = you protect your proprietary content.
Archives
archive.org_bot (Wayback Machine)
Public archives. Useful for brand history.
SEO scrapers (competing tools)
SemrushBot · AhrefsBot · MJ12bot · DotBot · DataForSeoBot · BLEXBot · PetalBot
Tools your competitors use to spy on your SEO. They consume your bandwidth without sending you a single visitor. Blocked by default in Livada SEO.
Opaque scrapers
Anchor Browser
Bots with an opaque identity, often used as relays by data scrapers.
6 Example of generated robots.txt
Here's an excerpt of the robots.txt
that Livada SEO generates automatically (33 blocks total, 1 per bot):
User-agent: *
Allow: /
Disallow: /wp-admin/
Sitemap: https://yoursite.com/sitemap.xml
# Livada SEO — AI bot access control
User-agent: Googlebot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
# ... 23 other allowed bots ...
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
# ... other blocked SEO scrapers ...7 What if my CDN still blocks after configuration?
If the live diagnostic returns 403 on a bot you've allowed in Livada SEO, the block is coming from the edge (CDN or host WAF), not WordPress. Here's where to check depending on your stack:
Dashboard → your domain → AI Crawl Control (or Security → Bots → AI Audit). Check that the "Block AI crawlers" toggle is set to Do not block, or uncheck individual bots to allow them.
hpanel → Security → Bots. If you're using Hostinger's automatic Cloudflare integration, control happens in the CF partner account — to take back control, migrate the domain to your own Cloudflare account (personal nameservers).
Manager → IP / Anti-DDoS → check that no managed WAF rule is blocking AI user-agents.
8 Frequently asked questions
What's the performance cost of strict blocking? ▾
Negligible. The hook runs on parse_request (priority 1)
before any theme, heavy plugin, or DB query loads. The logic does a
simple user-agent string comparison against the registry (33 entries max).
Typical measurement: <0.1 ms per request.
Does this block real users? ▾
No. Matching is done on very specific bot user-agents
(e.g., SemrushBot/7). No real browser contains these strings.
Verified on 5 production sites: 0 false positives.
What happens if I deactivate the plugin? ▾
The enriched robots.txt reverts to the WordPress default and the PHP hooks are removed. No residual trace. The plugin is non-invasive.
How do I know if my pages are being cited by AI today? ▾
Livada SEO's live diagnostic tells you whether technical access is possible (200 vs 403). To measure actual citation, you can (1) check your server logs (lines with an AI user-agent), (2) ask ChatGPT / Perplexity / Claude a question whose answer should involve your brand, (3) set up tracking via the plugin's "AI Visibility Score" page (0-100 score).
What other SEO plugins offer this feature? ▾
As of this doc's writing (May 2026): none. Yoast SEO, Rank Math, SEOPress, AIOSEO expose neither a detailed list of AI bots, nor live diagnostics, nor strict HTTP blocking. It's a Livada SEO exclusive.
What about Cloudflare's managed robots.txt? ▾
Cloudflare can serve its own managed robots.txt ahead of WordPress's (visible
via a # BEGIN Cloudflare Managed content comment).
In that case, the Livada SEO block is added after it. If Cloudflare's
directive contradicts your Livada SEO choice, CF wins at the edge level.
Solution: go to Cloudflare's AI Crawl Control panel to align them.
Ready to see if AI can reach your site?
Install Livada SEO 1.12+ and run the live diagnostic. 30 seconds to find out if your pages are visible to ChatGPT, Claude, Perplexity, Gemini and Copilot.