What is Crawl Budget

Why it matters

Crawl budget only becomes a real constraint on large sites — roughly tens of thousands of URLs and up — but on those sites it is decisive. If Googlebot spends its limited crawl allowance on faceted-filter permutations, session-ID URLs, or infinite calendar pages, your genuinely important pages get crawled and refreshed less often, so new content is discovered slowly and updates take longer to rank. For a small brochure site it is a non-issue; for a large e-commerce or publisher site it directly caps how fast the catalogue can be indexed and kept fresh.

• Matters most on large sites (tens of thousands of URLs+)
• Wasted crawl means slow discovery of new pages and slow refresh of updates
• Rarely a problem for small sites — do not over-engineer it there

How to protect it

Crawl budget is protected by not manufacturing low-value URLs in the first place, and by steering Googlebot away from the ones you cannot avoid. The two levers are crawl control (robots.txt, parameter handling) and index control (canonicals, noindex) — they do different jobs and are often confused. A page blocked in robots.txt is never crawled; a noindexed page is crawled but kept out of the index.

• Block infinite/parameter URL spaces (faceted filters, sort orders) from crawling
• Consolidate duplicates with canonical tags so equity is not split
• Fix internal links pointing to redirects and 404s — every hop wastes budget
• Keep the XML sitemap clean so Google prioritises the URLs you actually want

Worked example

A retailer with 200,000 products sees new stock taking weeks to appear in Google. A log-file analysis reveals two-thirds of Googlebot's requests are hitting faceted-filter URLs — colour, size and sort combinations that generate near-infinite duplicate pages — while genuine product pages are crawled only every few weeks. The remedy is layered: disallow the filter-parameter patterns in robots.txt so bots stop crawling them, add canonical tags on any that slip through to consolidate their signals, fix internal links that point through redirect chains, and prune the sitemap to only live, canonical product URLs. Within a crawl cycle, Googlebot reallocates its budget to the real catalogue and new products start indexing in days. The principle: crawl budget is not something you 'increase', it is something you stop wasting.