serps.io

Crawl Budget

How many pages a search bot will crawl on your site.

Crawl budget is how many URLs a search engine is willing and able to fetch from your site in a given window. Google describes it as the combination of two things: a crawl capacity limit, set by how much your server can take without slowing down, and crawl demand, set by how much the engine thinks your URLs are worth revisiting. You cannot read the budget off a dashboard, but you can observe it. Search Console's crawl stats report shows requests per day with response codes and purpose, and your own server logs, filtered to verified Googlebot requests, show exactly which paths are consuming the fetches. The limitation worth stating plainly is that crawl budget is the wrong explanation for most indexing problems. Sites with a few thousand URLs effectively never hit a ceiling, so if pages are not indexed the cause is usually duplication, thin content, or weak internal linking rather than a budget. The figure is also site-wide and does not tell you why any single page was skipped. Where it does bite, it bites hard. On a store with 480,000 crawlable URLs, logs showed Googlebot making 92,000 requests a day with 61 percent landing on faceted filter combinations that returned near-duplicate listings, and only 7 percent reaching product pages. Blocking three parameters and removing them from internal links moved product requests from 6,400 to 26,000 a day, and new products started getting indexed within two days instead of three weeks. The 2026 wrinkle is extra traffic from AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot, which hit the same wasteful URL patterns. Tidying crawl paths now reduces load and improves what both search engines and answer engines actually see.

Start ranking without the enterprise tax

Every plan starts with a 7 day trial. Connect Search Console and see where you stand in minutes.

Start your trial

7 day trial, card required, cancel any time.