Free tool · runs in your browser
robots.txt for AI Crawlers
Not every AI bot wants the same thing: some collect training data, others index for AI search and cite you, others fetch a page live for a user. Blocking everything wholesale means disappearing from AI answers. The generator separates the three groups cleanly.
robots.txt Generator
Allow or block AI crawlers selectively
Toggle whole bot groups or individual crawlers, copy the block. Everything runs in your browser.
Honest note: If you want to show up in AI answers, blocking the "Search & answers" group is counterproductive – that's why it's allowed by default. Whether you block training on your content is a business decision.
Training
Search & answers
User requests
Your selection never leaves the browser – the rules are generated locally via JavaScript.
Everything is generated locally in your browser. Nothing you enter is transmitted to our server.
How it works
- 1 Decide per group: allow or block
- 2 Override individual bots if needed
- 3 Paste the block into your existing robots.txt
Guides for this tool
- Visible in ChatGPT & AI Overviews: GEO for Your Website AI search is cutting organic clicks hard. What Generative Engine Optimization (GEO) is, what demonstrably helps – and what we do on our own website.
- llms.txt in Practice: What It Does and How We Build It An honest look at llms.txt: what the proposal promises, what is actually proven – and how to generate the file from your CMS instead of maintaining it.
Related Services
More tools
- wp-config.php Generator
- .htaccess Password Protection Generator
- Redirect Generator (301)
- PHP Limits Generator
- Restore WordPress Admin Access
- Cron Generator for WordPress
- SQL Generator: WordPress Database Cleanup
- Database Connection Test
- Caching Generator (.htaccess / Nginx)
- llms.txt Generator
- Maintenance Page Generator
- Backup Planner
- Base64 Encode and Decode
- HTTP Status Checker
- All tools →