AI search
AI Crawler Access Checker
Which AI crawlers your robots.txt lets in
Enter a domain and get a row for each AI crawler, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and others, marked allowed or blocked, with the robots.txt line that decided it and what that agent is actually used for.
Get early accessWhat it returns
- Allowed or blocked for each named AI user agent, with the robots.txt line that decided it quoted by line number.
- What each agent is for, stated separately: crawling for model training, fetching a page to answer a question right now, and collecting a public dataset are three different exposures.
- Google-Extended on its own row, because it governs Gemini grounding and AI training and has no effect on Googlebot or on whether you appear in Google Search.
- The agents that fell through to your wildcard group, so a blanket rule written years ago to stop scrapers is visible as the thing now blocking an answer engine.
When this matters
Someone adds a block for GPTBot after a board conversation about AI scraping, and a year later nobody can say what the site allows. The list is much longer than one agent and the agents do different jobs: one crawls to train a model, another fetches your page in order to answer a question that was asked a second ago, a third collects a public crawl that other models are built from. Blocking any one of them does nothing to the others, and blocking a retrieval agent quietly removes you from answers you could have been cited in.
How to answer this today
AI visibility measures the outcome rather than the permission: it runs your prompts against several answer engines, samples each one more than once because the engines are not deterministic, and records which domains got cited. That tells you whether you show up. Reading your robots.txt agent by agent tells you whether you are reachable at all, which is the first thing to rule out when you do not.
AI visibilityWhy this is harder than it looks
Blocking one does not block the rest
Each vendor runs its own agents, sometimes several with different purposes, and the names change as products do. A rule naming one bot is a rule about one bot. There is no single switch, and any tool that implies otherwise is giving a false sense of control in both directions.
Precedence is by specificity
Google evaluates the most specific matching rule rather than the first one in the file, so a long Allow path can override a short Disallow written below it. Group selection is separate again: when a group names an agent, that group applies and the wildcard group is ignored for it entirely.
robots.txt is a request
It is honoured by the crawlers that choose to honour it and is not enforcement. Blocking at the edge by user agent or network is a different control with different consequences, and some fetches are an assistant acting on one user's instruction rather than a crawler, which the file was never designed to describe.
Blocking has a cost on both sides
Training and retrieval are separate decisions. Refusing training crawls keeps your text out of a future model. Refusing retrieval crawls keeps your page out of the answer being written today, including the citation and the link. Treating them as one policy means accepting a consequence nobody chose.