Technical

Robots Meta Tester

See every robots directive a URL actually serves

In build

Enter a URL and get the robots meta tag and the X-Robots-Tag header exactly as served, every directive each one carries, which of them conflict, and whether the page is crawlable enough for a noindex to be read at all.

Get early access

What it returns

The robots meta tag from the HTML head and the X-Robots-Tag HTTP header, quoted as served, with every directive each one carries.
The directives in force once both sources are combined, because a noindex in the header applies even when the meta tag says index.
Whether robots.txt allows the URL to be crawled, since a noindex on a disallowed URL is never fetched, never read and therefore never obeyed.
Per-agent directives kept separate, so a rule aimed at one named crawler is not reported as applying to all of them.

When this matters

A page you wanted out of the index is still listed months after someone added a Disallow line for it. Blocking a crawler does not remove a page: Google can keep a URL in results on the strength of external links alone, usually with no snippet, because it never fetched the page to see the noindex waiting inside. The fix is the opposite of what it looks like. You have to let the crawler back in so it can read the directive that removes the page.

How to answer this today

Site audit reports this across the whole site: the pages carrying noindex, the pages blocked in robots.txt, and the pages where those two instructions contradict each other, found by crawling rather than by anyone remembering to look. What it does not do is read one URL on demand, which is what you want when a page has just shipped and you need to know what it is telling crawlers right now.

Site audit

Why this is harder than it looks

Blocking crawling is not blocking indexing

robots.txt controls fetching, nothing else. A disallowed URL can still be indexed from links on other sites and shown without a snippet. Only noindex reliably keeps a page out, and noindex only works on a page the crawler is allowed to fetch, so the two instructions together cancel each other out.

Two sources, one outcome

Directives arrive in the HTML head and in an HTTP header, and the header is often set at the CDN or server rather than in the template, which makes it invisible to anyone reading the page source. When the two disagree the restrictive one applies, so reading only the markup gives a confident wrong answer.

Not every directive does what its name suggests

none expands to noindex and nofollow together, several older directives were retired and are now silently ignored, and some are honoured by one search engine and not another. A tester is only useful if it says which directives had an effect rather than listing everything it found.

Directives added by JavaScript

A robots meta tag injected after the page loads is not in the served HTML, so a plain fetch reports no directive where a rendered page has one. The reverse also happens, with script removing a tag that the raw response contains. A reading has to say whether it rendered the page.