About the Robots.txt Generator
This robots.txt generator builds the plain-text file that tells search engine crawlers which parts of your site they may request. Choose whether to allow or block everything by default, list the paths to disallow (and any exceptions to allow), add your XML sitemap URL, and optionally block well-known AI training crawlers. Copy the result and save it as robots.txt in the root of your domain, for example https://example.com/robots.txt.
It is meant for site owners, SEO specialists and developers setting up a new site, keeping admin, cart, internal search or staging paths out of crawl queues, or opting out of AI data collection. Paths are cleaned up automatically — leading slashes are added and duplicates removed — so the output follows the Robots Exclusion Protocol syntax.
Remember that robots.txt controls crawling, not indexing: a disallowed URL can still appear in search results if other sites link to it, and the file is public, so never use it to hide sensitive pages. Use a noindex meta tag or authentication for that. Compliance is voluntary; reputable crawlers obey it, malicious bots may not.
How to use the robots.txt generator
- 1Choose whether crawlers may access the site by default.
- 2List the paths to keep crawlers out of, one per line, and any exceptions to allow.
- 3Optionally block AI training crawlers and set a crawl-delay.
- 4Enter the full URL of your XML sitemap.
- 5Copy the output, save it as robots.txt and upload it to your site’s root folder.
Formula and method
The output follows the Robots Exclusion Protocol (RFC 9309). A group starts with one or more User-agent lines and contains Allow and Disallow rules matched against the start of the URL path. When rules conflict, Google and other major crawlers apply the most specific (longest) matching rule, and Allow wins a tie — which is why exception paths can be listed alongside broader Disallow rules.
The generator writes one group for all crawlers (User-agent: *) with your allow exceptions and disallowed paths, an optional Crawl-delay, an optional second group that disallows the listed AI crawlers from the whole site, and a Sitemap line, which applies to the whole file regardless of group. An empty rule set is written as Allow: / so the file explicitly permits crawling.
Worked examples
Online store hiding admin, cart and search
One User-agent: * group with four Disallow rules (/admin/, /cart/, /checkout/ and /search) plus the Sitemap line gives a six-line file that keeps low-value pages out of crawl queues.
Staging site blocked entirely
Blocking the whole site produces the classic two-line file: User-agent: * and Disallow: /. No sitemap is listed because the site should not be crawled.
Block AI crawlers with an allowed exception and crawl-delay
The main group has one Allow exception, four Disallow rules and a Crawl-delay of 10. A second group lists six AI crawler user-agents with Disallow: /, and the Sitemap line follows — 15 non-blank lines and 6 rules in total.
Frequently asked questions
Where do I put robots.txt?+
It must sit at the root of the host it applies to, such as https://www.example.com/robots.txt. Each subdomain and protocol needs its own file, and crawlers will not look for it in subfolders.
Does robots.txt stop a page from being indexed?+
No. It stops compliant crawlers from fetching the page, but the URL can still be indexed from links elsewhere, usually without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or HTTP header.
How do I block AI crawlers like GPTBot?+
Add a group naming the crawler’s user-agent token followed by Disallow: /. Tokens include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google AI training), Applebot-Extended (Apple AI training) and PerplexityBot (Perplexity’s search crawler). Turn on the AI option to add them automatically.
Does Google support crawl-delay?+
No. Googlebot ignores the Crawl-delay directive and adjusts its crawl rate automatically based on how your server responds. Bing, Yandex and some other crawlers do honour it.
What does an empty Disallow mean?+
A Disallow line with no path (Disallow:) means nothing is disallowed — the same as Allow: /. A Disallow: / line, by contrast, blocks the entire site, so check that single slash carefully.