robots.txt & llms.txt (AI Crawlers)
Control search engine crawl budgets with a smart robots.txt generator, then publish an llms.txt policy so AI systems like GPTBot and ClaudeBot can discover your best content.
Before you start
Crawl Intelligence is a two-tab workspace under Tools & Analytics. The robots.txt tab manages search engine crawl rules; the llms.txt tab publishes an AI content access policy at /llms.txt.
You will need:
- WordPress Administrator access
- No conflicting physical robots.txt file in your site root (plugin serves it dynamically)
- Permalinks enabled (pretty URLs) for
/robots.txtand/llms.txtto resolve
Open Crawl Intelligence
Navigate to:
SEO Rank Genius â Tools & Analytics â Crawl Intelligence
The section has two tabs in the Tools sidebar:
robots.txt Smart mode
Smart mode generates a recommended robots.txt with one click. It blocks low-value paths (admin, feeds, search results) while allowing your sitemap and important content.
robots.txt
Publish a search-friendly robots.txt at yoursite.com/robots.txt
Block Sections
robots.txt tab â Smart/Advanced mode selector, Disallow/Allow KPIs, and section checkboxes.
Toggle Enable in the header to publish the file. Smart mode is the right default for most WordPress sites.
robots.txt Advanced
Switch to Advanced mode for full manual control: add custom Allow/Disallow rules, set crawl-delay directives, and manage user-agent-specific blocks.
Custom Rules
Live Preview
Advanced mode â custom Allow/Disallow rules with live code preview.
yoursite.com/robots.txt in a private browser window and confirm the preview matches the live file. Clear any CDN or full-page cache first.llms.txt policy
The llms.txt tab publishes a machine-readable content policy at /llms.txt. AI systems use it to understand which pages represent your authority content and how they may use your material.
llms.txt – AI Content Access Policy
Help AI systems discover your best content
Markdown Preview
llms.txt tab â pillar pages KPI, crawler count, and Markdown preview at /llms.txt.
Enable llms.txt v2 PRO for authority-scored, llmstxt.org-compliant output that prioritizes pillar-cluster content structure.
AI crawler access
Individual toggles control which AI bots may access your content. Each bot has a dedicated user-agent recognized by major AI platforms.
| AI Crawler | Platform | Access |
|---|---|---|
| GPTBot | OpenAI / ChatGPT | |
| ClaudeBot | Anthropic / Claude | |
| PerplexityBot | Perplexity AI | |
| Google-Extended | Google Gemini |
AI crawler rows â per-bot access toggles and priority pillar pages option.
Troubleshooting
/robots.txt shows WordPress default or 404
Enable the robots.txt toggle in Crawl Intelligence and save. Flush permalinks under Settings â Permalinks (click Save without changes). Remove any physical robots.txt file from your server root.
Preview differs from live file
Purge CDN and page cache. Another SEO plugin may also be serving robots.txt â disable its robots feature to avoid conflicts.
/llms.txt returns 404
Enable llms.txt in the tab header, save settings, and flush permalinks. Confirm pretty permalinks are active (not Plain).
AI crawler still blocked after enabling
Check robots.txt for a blanket Disallow: / on the bot user-agent. Smart mode handles AI bots separately â re-save in Smart mode or add an explicit Allow rule in Advanced.
Staging environment blocks everything
This is intentional. The staging detection banner appears at the top of the robots.txt tab. Only override on production domains.