Robots.txt Generator

What a robots.txt file actually does

A robots.txt file is a plain text file at the root of a host that tells crawlers which URLs they may request. It is a request, not a lock. Every well-behaved search engine reads it before fetching anything else, obeys the rules it finds, and caches the file for up to a day. Every badly behaved scraper reads it too, sometimes as a map of the directories you would rather nobody saw. That is the whole security model, and it is why the file should never be the thing standing between the public and your customer data.

The syntax is small. A group starts with one or more User-agent lines naming the crawler, followed by Disallow and Allow lines giving URL path prefixes. A crawler obeys exactly one group: the one whose user-agent token matches it most specifically, falling back to the * group. An empty Disallow: means nothing is blocked. Disallow: / means everything is. The Sitemap line is independent of any group and can appear anywhere in the file.

Worked example: an online shop

Say you run store.example.com on Shopify or WooCommerce. You want product and category pages crawled and everything transactional left alone, because faceted URLs and session-based cart links generate an effectively infinite crawl space that burns budget without ever ranking. A realistic file looks like this:

Note the ordering rule that trips people up: within a group, Google applies the most specific matching rule, not the first one. A page at /products/summer-sale?sort=price is matched by both Disallow: /*?sort= (8 characters of pattern) and Allow: /products/ (10 characters), so the longer Allow wins and the page stays crawlable. If two rules are the same length, Allow wins. This is why blanket wildcards are safer once you pair them with explicit Allow lines.

Crawling is not indexing

This is the single most expensive misunderstanding in technical SEO. Disallowing a URL stops the crawler downloading it; it does not remove the URL from the index. If any page anywhere links to that URL, Google can and will list it, showing the bare URL with the note "No information is available for this page". Worse, because the crawler is forbidden from fetching the page, it can never see the noindex tag you put there, so the block actively prevents the removal you wanted.

The correct sequence for de-indexing an already-indexed page is: allow crawling, serve <meta name="robots" content="noindex"> or an X-Robots-Tag header, wait for a recrawl, confirm the page has dropped out, and only then add a Disallow if you also want to save crawl budget. Use robots.txt for crawl management — infinite parameter spaces, internal search results, staging assets, print views. Use noindex for index management. They solve different problems.

Deciding about AI crawlers

Since 2023 a second category of user agent has mattered: bots that collect text for training or for answering questions in a chat interface. GPTBot (OpenAI training), ClaudeBot (Anthropic), Google-Extended (Gemini training, separate from Googlebot's search crawl), CCBot (Common Crawl, which feeds many datasets), PerplexityBot and Bytespider (ByteDance) all publish tokens you can name in a group. Blocking them is one line each.

The tradeoff is genuine and worth thinking about rather than copying someone's opinion. Blocking protects content you sell or license, and it keeps your writing out of training corpora. It also removes you from the answer engines that a growing share of readers now use instead of a search page, and citation traffic from those surfaces is real. A common middle position is to block training crawlers such as GPTBot, Google-Extended and CCBot while allowing retrieval bots that cite and link back. Whatever you choose, remember that Google-Extended does not affect your normal search ranking — it governs Gemini training only, and blocking it costs you nothing in classic search results.

Testing, limits and common mistakes

Once uploaded, open https://yourdomain.com/robots.txt in a browser first — a surprising number of sites return an HTML 404 page there, which crawlers treat as "no restrictions". Then use the robots.txt report in Google Search Console to see the fetched version, its status and any parse errors, and Bing Webmaster Tools for the same on Bing's side. Both show you the file as the crawler sees it, which is what matters when a CDN is caching an old copy.

Limits worth knowing: Google reads only the first 500 KiB and ignores the rest; a 5xx response for longer than about 30 days is treated as if the file did not exist; the file applies per host and per protocol, so https://example.com and https://www.example.com need their own copies. Crawl-delay is not part of Google's implementation at all. And the classic disaster remains a staging Disallow: / shipped to production — check that line before every launch, because search traffic can take weeks to recover.