The Definitive robots.txt + llms.txt + Cloudflare WAF Guide for AI Crawlers (2026)
34% of SaaS websites accidentally block at least one major AI crawler via Cloudflare Bot Fight Mode or misconfigured robots.txt rules. This guide gives you the exact configuration to unblock every major AI search engine —” and the llms.txt template to maximize citation yield.
1. The AI Crawler Landscape (2026)
Seven distinct AI crawlers need access to your site. Each serves a different AI engine and has different implications for your brand's citation visibility:
| Crawler User-Agent | AI Engine | Priority |
|---|---|---|
| GPTBot | ChatGPT (training) | —”— Critical |
| OAI-SearchBot | ChatGPT Search | —”— Critical |
| PerplexityBot | Perplexity AI | —”— Critical |
| ClaudeBot | Claude / Anthropic | — High |
| Google-Extended | Gemini / AI Overviews | — High |
| Applebot-Extended | Apple Intelligence | 🟡 Medium |
| bingbot | Bing AI / Copilot | 🟡 Medium |
2. The Complete robots.txt Configuration
Replace or augment your current robots.txt with this production-ready configuration. The critical rule is to explicitly Allow: / for every AI crawler —” a missing entry means the crawler falls back to whatever your wildcard rule says:
# ============================================================ # robots.txt —” AI Crawler Access Configuration # Last updated: October 2026 # ============================================================ # ── Traditional Search Engines ────────────────────────────── User-agent: Googlebot Allow: / User-agent: bingbot Allow: / User-agent: Slurp Allow: / # ── AI Training Crawlers ───────────────────────────────────── User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / # ── Gemini / Google AI ─────────────────────────────────────── User-agent: Google-Extended Allow: / # ── Apple Intelligence ─────────────────────────────────────── User-agent: Applebot-Extended Allow: / # ── Optional: Block AI Training Only (Keep Search) ────────── # If you want Perplexity to search-cite you but NOT use your # content for AI training, comment out PerplexityBot Allow above # and replace with: # User-agent: PerplexityBot # Disallow: /private/ # (Most sites should Allow: / for all) # ── Global Fallback ────────────────────────────────────────── User-agent: * Allow: / Disallow: /api/private/ Disallow: /admin/ Disallow: /_next/ # ── Sitemap Declaration ────────────────────────────────────── Sitemap: https://yourdomain.com/sitemap.xml Sitemap: https://yourdomain.com/llms.txt
3. Cloudflare WAF Bypass Rules (The #1 Overlooked Fix)
— ️ Critical: Cloudflare's Bot Fight Mode and Super Bot Fight Mode block crawlers based on IP reputation and behavioral fingerprinting —” not robots.txt. Even with a perfect robots.txt, Cloudflare will serve AI crawlers a CAPTCHA or 403 if they trigger bot detection. Your robots.txt fix does nothing without this WAF step.
Step-by-Step: Cloudflare WAF Custom Rule
Go to your Cloudflare dashboard —’ Security —’ WAF —’ Custom Rules
Click 'Create Rule'
Name it: 'AI Crawler Bypass —” Allow All AI Bots'
Set the expression (use Expression Editor for complex OR chains):
(http.user_agent contains "GPTBot") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "Applebot-Extended") or (http.user_agent contains "Amazonbot")
Action: Skip —’ Skip all remaining custom rules. This tells Cloudflare to pass these crawlers through without applying Bot Fight Mode, rate limiting, or any other WAF rules. Do NOT use “Allow” —” use “Skip” to bypass the entire rule chain.
4. Building Your llms.txt File
/llms.txt is the emerging standard for giving AI crawlers a curated, Markdown-formatted map of your site. It's similar to robots.txt but instead of controlling access, it guides understanding. Here's the production template:
# [Your Company Name] > [One sentence describing what your company does and who it's for] [Your Company] is a [category] platform that [primary value proposition]. Founded in [year], serving [X] customers across [industries/geographies]. ## Product Pages - [Product Name](https://yourdomain.com/product): [One line description] - [Feature A](https://yourdomain.com/features/a): [One line description] - [Feature B](https://yourdomain.com/features/b): [One line description] ## Pricing - [Plan A - $XX/month](https://yourdomain.com/pricing): [Who it's for] - [Plan B - $XX/month](https://yourdomain.com/pricing): [Who it's for] ## Documentation - [Getting Started](https://yourdomain.com/docs): Setup guide - [API Reference](https://yourdomain.com/docs/api): Full API docs ## Key Resources - [Blog](https://yourdomain.com/blog): Technical guides and playbooks - [FAQ](https://yourdomain.com/faq): Common questions answered - [Case Studies](https://yourdomain.com/customers): Customer success stories ## Entity Information - Wikipedia: [URL if exists] - Wikidata: [Q-item URL] - LinkedIn: [Company page URL] - Crunchbase: [Profile URL] - G2: [Profile URL]
—’— Pro tip: After publishing your llms.txt, reference it in your sitemap.xml (as a supplemental resource) and add a link to it from your robots.txt Sitemap declaration. This maximizes AI crawler discovery speed.
5. Verification: Did It Work?
After implementing all three layers, verify access is working:
Check If AI Crawlers Can Access Your Site
Optymia's Crawler Intelligence module simulates GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot against your domain —” showing you exactly which pages are blocked and the precise WAF or robots.txt rule causing each block.