Technical

Robots.txt Tester

Confirm what crawlers can and cannot reach

Coming soon

Paste a robots.txt and a URL to check whether a given user agent is allowed to crawl it, with the matching rule highlighted.