Technical
Robots.txt Generator
Write the file before it blocks the wrong thing
Pick the paths and crawlers you want to keep out and get a complete robots.txt: grouped user-agent blocks, allow and disallow lines in precedence order, your sitemap declarations, and a note on every rule that will do nothing.
Get early accessWhat it returns
- A complete robots.txt with one group per user agent, ordered the way a parser reads it rather than the order the rules were thought of.
- Allow and disallow lines for the paths you named, with wildcards and the end-of-URL anchor written out, so each rule matches what you meant by it.
- Your sitemap declarations as absolute URLs, which is the one directive that belongs to the file rather than to a user-agent group.
- The lines that will be ignored: crawl-delay for crawlers that do not honour it, noindex in robots.txt, and rules already covered by a more specific one.
When this matters
A site is built on a staging host behind a blanket disallow, DNS cuts over on a Friday, and the file goes with it. Two weeks later nobody can explain why the new pages are not indexed. The opposite failure is quieter: a well-meant rule to hide an internal search path, written with no anchor, also blocks every URL beginning with those characters, including a category that was earning traffic. Generating the file from the paths you actually want blocked makes both mistakes visible before they ship.
How to answer this today
Site audit reads the robots.txt that is live and reports the URLs the crawl was not allowed to reach, which is how a mistake already in production gets found. It works from the deployed file, so it can tell you a rule is costing you pages but cannot help you write the file in the first place.
Site auditWhy this is harder than it looks
A generator cannot know your site
Blocking by path only works if the paths are right, and the directory holding faceted URLs on one platform holds the entire catalogue on another. Output has to be read against the real URL structure, never pasted on faith.
Crawling is not indexing
A disallowed URL can still appear in results, because robots.txt stops the fetch and not the listing. Keeping a page out of search needs a noindex the crawler is allowed to read, which is the opposite of blocking it.
One file per host and scheme
Every host, subdomain and scheme serves its own robots.txt from its own root. Writing one file and deploying it everywhere is how a rule meant for the www version ends up governing nothing.
The rules are not uniform
Pattern support, group matching and the handling of unknown directives differ between crawlers, and AI crawlers in particular arrive under names that keep changing. A file that reads correctly for one agent can mean nothing to the next.