The Definitive Guide to robots.txt & llms.txt for AI Search Engines (2026)
Over 42% of websites that complain of zero citations in ChatGPT Search and Perplexity have misconfigured their robots.txt, blocked user-agents via Cloudflare Bot Fight Mode, or failed to publish an llms.txt index. Here is the complete production blueprint.
1. The 2026 AI Crawler Taxonomy: Who Is Knocking on Your Door?
In 2024, webmasters viewed AI bots as a monolithic threat scraping content without attribution. By 2026, the ecosystem has fractured into two distinct classes: Model Pre-training Crawlers (which vacuum raw text for foundation models) and Live Search & Retrieval Crawlers (which fetch real-time answers and generate high-intent outbound clicks to your domain).
| User-Agent | Operator | Primary Purpose | Impact if Blocked | Recommended Rule |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT Search live indexing | Zero ChatGPT citations | Allow |
| GPTBot | OpenAI | Foundation model pre-training | Excludes training data | Disallow or Allow |
| PerplexityBot | Perplexity AI | Perplexity Pro real-time RAG | Zero Perplexity citations | Allow |
| ClaudeBot | Anthropic | Claude search & artifacts retrieval | Excluded from Claude citations | Allow |
| Google-Extended | Gemini & Vertex training control | Blocks Gemini, NOT Google Search | Allow for Gemini AEO | |
| Applebot-Extended | Apple | Apple Intelligence generative search | Excluded from Siri/Apple answers | Allow |
2. The Production-Grade robots.txt File for 2026
Below is the standard robots.txt pattern implemented across enterprise sites using Optymia. It grants full access to search and retrieval bots while safeguarding sensitive admin panels, internal APIs, and auth gates:
3. Why Every Brand Needs an llms.txt File
Large Language Models operate on strict token budgets. When a user asks Perplexity or SearchGPT a question like "What is the best AI search tracker for Shopify stores?", the model doesn't want to scrape 50MB of CSS, unminified JavaScript bundles, and header tracking pixels.
The llms.txt standard provides an ultra-clean, Markdown-only interface. AI engines inspect this file to build an instant mental model of your architecture:
Structure of /llms.txt:
- H1 Title: Brand name and core category statement.
- Blockquote: Concise 2-sentence mission summary (direct snippet candidate).
- Core Capabilities: Bullet points specifying exact features, technical specs, and pricing tiers.
- Canonical Documentation: Clean Markdown links to programmatic solutions, developer guides, and comparative benchmarks.
Test Your Domain's AI Crawler Readiness
Enter your website URL below to test if GPTBot, PerplexityBot, and Google-Extended are granted access, inspect schema density, and calculate your live Astra Score.
Test your domain for AI search citations & Share of Model
Simulate how ChatGPT, Gemini, Perplexity, and Claude perceive your website.
4. Fixing Cloudflare & WAF False Positives
If your domain is behind Cloudflare, standard Bot Fight Mode frequently flags PerplexityBot and OAI-SearchBot as scraping attacks. To resolve this:
- Log into your Cloudflare Dashboard and navigate to Security > WAF > Custom Rules.
- Click Create Rule and name it "Allow Verified AI Search Engines".
- Set the rule expression to:
(http.user_agent contains "PerplexityBot" or http.user_agent contains "OAI-SearchBot" or http.user_agent contains "ClaudeBot") - Set Action to Skip > Select "All remaining custom rules" and "Bot Fight Mode".
- Deploy the rule. Verification will show 100% crawl success in your server logs within minutes.