Robots.txt Generator & Tester
Create a robots.txt from user-agent groups and Allow/Disallow rules, check a file you already have for mistakes, and test any URL against it to see whether it is allowed and which rule decided. Everything runs on this page; the tool never fetches your site.
/ and match from the start of the URL path. * matches any characters, $ marks the end of the URL, and an empty Disallow allows everything.Lint
Is this URL allowed?
About the AI-training preset
- It lists tokens their operators document for collecting AI training data: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), meta-externalagent (Meta), plus Google-Extended and Applebot-Extended, which are not separate crawlers but tell Google and Apple not to use pages fetched by their normal crawlers for model training.
- Search crawlers for AI answers (such as OAI-SearchBot or PerplexityBot) and fetchers that run when a user asks an assistant to open a page are different bots. They are not in the preset; add them yourself if you want. Operators say user-triggered fetches may not follow robots.txt.
- Blocking Google-Extended does not affect Google Search ranking or inclusion.
Good to know
- robots.txt is a voluntary request, not access control. Well-behaved crawlers obey it; anyone else can still fetch your pages. Protect private content with a login.
- Disallow stops crawling, not indexing: a blocked URL can still appear in results if other sites link to it. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header.
- Crawl-delay is not part of the standard. Google ignores it; Bing honours it.
- The file must live at the root of each host and protocol, e.g. https://example.com/robots.txt.
How to use
- Pick a preset — Allow everything, Block AI training crawlers, Block admin & search paths, or Block everything — or start from the default group.
- Edit each group: list its user-agents (one per line, * for every crawler) and add Allow or Disallow rules with paths such as /private/ or /*.pdf$.
- Add your sitemap URL so crawlers can find it, then copy or download the robots.txt and upload it to the root of your site.
- Use "Is this URL allowed?" to test a user-agent and URL: the result names the rule that decided, following RFC 9309 matching.
- Already have a file? Switch to Check an existing robots.txt, paste it to see lint warnings, test URLs against it, or load it into the builder.
Examples
User-agent: * Disallow: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent Disallow: / Sitemap: https://example.com/sitemap.xml
User-agent: Googlebot, URL: /fish.html under "Disallow: /fish" → BLOCKED by Disallow: /fish (line 2)
Supported formats
- Output: a plain-text robots.txt (UTF-8, no BOM) with User-agent groups, Allow/Disallow rules, optional Crawl-delay and Sitemap lines.
- Rule paths support the * wildcard and the $ end-of-URL anchor, as defined in RFC 9309 and Google's robots.txt specification.
Privacy
This tool runs entirely in your browser. Nothing you enter or open is sent anywhere. Privacy policy.
Limitations
- It does not fetch, crawl or upload anything: paste your existing file in by hand, and upload the generated one to your server yourself.
- robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it; use a noindex meta tag or header to keep a page out of results.
- robots.txt is voluntary — it is not security. Badly behaved bots can ignore it, so protect private content with authentication.
- The tester applies RFC 9309 exactly. Individual crawlers can add their own behaviour (for example, some fall back to a parent token's group), so real-world results can differ.
- The AI-training preset covers documented tokens only; new crawlers appear often, so review the list from time to time.
FAQ
How do I block GPTBot and other AI training crawlers?
Use the Block AI training crawlers preset. It adds a group for GPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended and Applebot-Extended with Disallow: /, while leaving search engines such as Googlebot and Bingbot allowed. Compliant operators honour it; it cannot stop crawlers that ignore robots.txt.
Will Disallow remove a page from Google?
No. Disallow only stops crawling. If the URL is linked from elsewhere, Google can still index it without seeing its content. To remove a page, let it be crawled and add a noindex meta tag or X-Robots-Tag header.
Which rule wins when Allow and Disallow both match?
The rule with the longest matching path wins. If an Allow and a Disallow match with paths of the same length, Allow wins. The tester shows exactly which rule decided.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl rate automatically. Bing and some other crawlers do honour it, so it can still be useful for them.