Robots.txt Generator
Build a valid robots.txt file with platform presets, sitemap lines and one-click AI crawler blocking.
How it works
Pick whether crawlers may index your whole site, list the paths you want kept out of
search results, and the generator writes a valid robots.txt file instantly.
Every line follows the Robots Exclusion Protocol that Google, Bing and the other major
search engines actually parse: a User-agent line starts a group, then
Disallow and Allow rules apply to that group, and
Sitemap lines sit on their own at the end of the file. Paths are normalised
for you — a missing leading slash is added, blank lines and duplicates are dropped — and
wildcards like /*.pdf$ are passed through untouched.
The platform presets fill in the paths that are usually worth blocking on WordPress,
Shopify, Drupal, Joomla and Magento (admin screens, internal search, cart and checkout
URLs). Two extra switches cover things most generators miss: one adds explicit blocks for
AI training crawlers such as GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot,
the other blocks the SEO scraper bots (AhrefsBot, SemrushBot, MJ12bot and friends) that
eat crawl budget without sending traffic. Remember that robots.txt controls
crawling, not indexing — to keep a page out of search results entirely use a
noindex meta tag on a page crawlers are still allowed to fetch.
Copy the result or download it, then upload it to the root of your domain so it is served
at https://yourdomain.com/robots.txt. Everything is generated in your browser
with JavaScript — no upload, no account, no ads, and your unreleased URLs never leave your
device.
Frequently asked questions
How do I create a robots.txt file?
Pick whether crawlers may index everything or nothing, list the paths you want kept out of search (one per line), optionally add your sitemap URL, then copy or download the result. The generator writes the User-agent, Disallow, Allow and Sitemap lines in the order search engines expect, adds missing leading slashes and removes duplicates, so the file is valid without you memorising the syntax.
Where do I put the robots.txt file on my site?
It must sit in the root of the domain and be served at https://yourdomain.com/robots.txt, because crawlers never look anywhere else, so a file in a subfolder is ignored. Each subdomain needs its own file, and it applies to that host only. Upload it over FTP or via your CMS, then open the URL in a browser to confirm it is served as plain text.
How do I block AI crawlers like GPTBot in robots.txt?
Tick "Block AI training crawlers" and the generator adds a separate group for each major AI bot (GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Applebot-Extended, Bytespider and more), each with Disallow: /. These crawlers honour robots.txt voluntarily, so it stops the well-behaved ones but is not a security control; anything truly private needs authentication.