All Tools
robots.txt Generator
Generate a clean robots.txt with explicit control over AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more).
Crawler controlAI bots
AI crawlers — block these
Leave all unchecked for maximum AI visibility. Only block if you want to opt out of being cited / used for training.
text
Sitemap: https://example.com/sitemap.xml User-agent: * Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /api/ # AI crawlers — all allowed (recommended for AI visibility)