How many pages a search engine will crawl on your site, and how often.
Crawl budget is the number of URLs Googlebot is willing and able to fetch from your site in a given period. It is a product of two things: crawl capacity (how fast your server responds without being overloaded) and crawl demand (how much Google wants your pages, driven by popularity and freshness).
For most small and mid-size sites it is not a concern. It becomes real for large sites (tens of thousands of URLs and up), sites with heavy faceted navigation, or sites generating many low-value URLs that waste crawls before important pages are reached.
If Google spends its budget crawling parameter URLs, filtered listings, or duplicate pages, your genuinely important content gets crawled and refreshed more slowly. That delays indexing of new pages and updates to existing ones.
The goal is to point crawlers at what matters and away from what does not.
Crawl budget only becomes a real constraint on large sites — roughly tens of thousands of URLs and up — but on those sites it is decisive. If Googlebot spends its limited crawl allowance on faceted-filter permutations, session-ID URLs, or infinite calendar pages, your genuinely important pages get crawled and refreshed less often, so new content is discovered slowly and updates take longer to rank. For a small brochure site it is a non-issue; for a large e-commerce or publisher site it directly caps how fast the catalogue can be indexed and kept fresh.
Crawl budget is protected by not manufacturing low-value URLs in the first place, and by steering Googlebot away from the ones you cannot avoid. The two levers are crawl control (robots.txt, parameter handling) and index control (canonicals, noindex) — they do different jobs and are often confused. A page blocked in robots.txt is never crawled; a noindexed page is crawled but kept out of the index.
A retailer with 200,000 products sees new stock taking weeks to appear in Google. A log-file analysis reveals two-thirds of Googlebot's requests are hitting faceted-filter URLs — colour, size and sort combinations that generate near-infinite duplicate pages — while genuine product pages are crawled only every few weeks. The remedy is layered: disallow the filter-parameter patterns in robots.txt so bots stop crawling them, add canonical tags on any that slip through to consolidate their signals, fix internal links that point through redirect chains, and prune the sitemap to only live, canonical product URLs. Within a crawl cycle, Googlebot reallocates its budget to the real catalogue and new products start indexing in days. The principle: crawl budget is not something you 'increase', it is something you stop wasting.
Part of our defined terms knowledge graph — browse every entry in this branch.
A root-level file that tells crawlers which URLs they may or may not fetch.
Whether a page is stored in a search engine’s index and eligible to rank.
A machine-readable list of your important URLs, submitted to help discovery.
A structured process for increasing the share of visitors who convert.
The share of people who click after seeing your ad or search result.
Common questions
Straight answers on how this fits your marketing and build.
Still have questions? Talk to a specialist