What Is a Crawl Budget Audit (and When You Actually Need One)

A crawl budget audit is the process of figuring out how Googlebot is spending its limited crawl requests on your site and where that budget is going to waste instead of reaching the pages you actually want indexed. Effective crawl budget optimization ensures that Googlebot focuses its efforts on your most valuable pages.

If you'd rather have a specialist run this end to end, our technical SEO audit process covers exactly what's outlined below. Crawl budget itself is made up of two parts. Crawl capacity is how many requests your server can handle without slowing down or throwing errors. Crawl demand is how much Google actually wants to crawl your pages, based on freshness, internal linking, and perceived value. An audit looks at both sides: is your server holding Google back, and is Google choosing to spend its time on the wrong pages?

Not every site needs this. If you're running a site with a few hundred pages and solid internal linking, crawl budget is almost never your bottleneck. Google can usually crawl a small site completely within a day or two, and your indexing problems, if you have any, are more likely about content quality or technical errors than crawl allocation. If that sounds like your situation, a broader technical SEO audit is usually the better starting point.

This audit is built for sites where crawl budget is a real constraint: large catalogs, sites with faceted navigation or URL parameters, content hubs publishing frequently, or JavaScript-heavy sites where rendering adds extra crawl cost. If you're managing 500+ pages, especially with dynamic URLs or frequent content changes, this is where the work below pays off and where our team typically gets involved.

Signs Your Large Site Has a Crawl Budget Problem

Before running a full crawl budget audit, check for these signals. They won't tell you the cause, but they'll tell you whether it's worth the time.

A large gap between pages published and pages indexed. If you've published 10,000 product pages but Search Console shows a fraction of that indexed, something is off.

A rising "Discovered but not indexed" or "Crawled but not indexed" count in the Index Coverage report. This means Google knows the URLs exist but isn't prioritizing them.

New content takes days or weeks to get crawled, even though you're publishing regularly and linking to it internally.

Organic traffic is flat or declining despite consistent content output. If you're publishing at a healthy pace but not seeing corresponding indexation or traffic growth, crawl allocation is worth investigating.

A fast way to sanity-check this is to compare the number of URLs in your XML sitemap against the number Search Console reports as indexed. If that ratio is well below what you'd expect for a healthy site, move on to the full audit, or confirm it's worth the deeper dive before committing more time.

The Crawl Budget Audit Framework

The audit breaks down into six steps, in order:

1. Diagnose in Google Search Console to get a high-level view of crawl activity and indexation status.

2. Analyze server log files to see exactly what Googlebot is doing on your site, not just what Search Console summarizes. This log file SEO technique provides a deeper understanding of crawl patterns.

3. Crawl the site with a tool like Screaming Frog to map your full URL structure, internal linking, and technical issues.

4. Cross-reference your sitemap, your crawl data, and your index status to find where crawl budget and indexation don't line up.

5. Prioritize fixes based on which issues affect your highest-value pages.

6. Monitor on an ongoing basis, since crawl budget isn't a one-time fix.

Each step builds on the last. Search Console tells you something is wrong. Log files tell you where. A full site crawl tells you why. The rest is about fixing the right things in the right order.

Step 1: Diagnose in Google Search Console

Start here because it's free, fast, and gives you a baseline before you go digging into logs.

Reading the Crawl Stats Report

In Search Console, go to Settings, then open the Crawl Stats report under the Advanced section. This shows:

• Total crawl requests over the last 90 days

• Average server response time

• A breakdown of requests by response code (200, 301, 404, 500, etc.)

• A breakdown by file type (HTML, images, JavaScript, CSS)

What to look for: a flat or declining crawl request trend despite the site growing is a warning sign. So is a response time that's crept upward since Google tends to slow its crawl rate when your server takes longer to respond. A high proportion of non-200 responses, especially 4xx and 5xx codes, means Google is spending requests on pages that don't return usable content.

Reading the Index Coverage Report

This report categorizes every URL Google knows about. The categories worth paying attention to during a crawl budget audit are:

A large and growing "Discovered, currently not indexed" count on a big site is one of the clearest signs that crawl demand isn't keeping up with the number of URLs you're generating.

Step 2: Analyze Server Log Files

This is where most crawl budget guides stop short. They'll tell you log analysis is the most accurate method and then move on without explaining how to actually do it. This is the step that tells you what's really happening because it's a direct record of every request Googlebot made, not an aggregated summary. If your team doesn't have log access set up yet, a dedicated technical SEO team can handle extraction, parsing, and Googlebot verification for you.

Getting Access to Your Logs

Depending on your hosting setup, you can usually pull logs from:

• Your server directly (Apache or Nginx access logs)

• Your CDN, if you use one (Cloudflare, Fastly, and similar all offer log access)

• A hosting panel or managed platform's log export feature

If you're on a platform where raw server logs aren't accessible, CDN logs are usually a workable substitute since most Googlebot requests pass through the CDN layer.

Verifying You're Looking at Real Googlebot Traffic

User agent strings can be spoofed, so don't filter on the "Googlebot" string alone. Verify requests by reverse DNS lookup on the IP address (it should resolve to a googlebot.com or google.com domain) or use Google's published IP ranges. Most log analysis tools handle this verification automatically, which is one reason to use one rather than filtering a raw log manually.

What to Extract From the Logs

Once you have verified Googlebot requests, look at:

• Most and least crawled URLs. Are your highest-value pages (money pages, top category pages) getting crawled frequently, or is budget concentrated on low-value sections?

• Crawl frequency by site section. Segment by URL pattern (/blog/, /products/, /category/) to see where crawl activity actually goes versus where it should.

• Status codes Googlebot is hitting. If a meaningful share of requests return 404s, 500s, or redirects, that's budget being spent on dead weight.

• Requests to non-indexable URLs. Parameter URLs, session IDs, or internal search result pages that shouldn't be crawled at all but are eating into your budget anyway.

Tools for Log Analysis

For sites with a few hundred thousand log lines, Screaming Frog's Log File Analyzer is a practical starting point and handles Googlebot verification automatically. For enterprise-scale sites with millions of URLs, dedicated platforms like Botify or JetOctopus combine log data with crawl and Search Console data in one place. If you're comfortable with spreadsheets or a query tool like BigQuery, a manageable log sample can be filtered and pivoted manually, though this gets unwieldy past a certain volume.

Step 3: Crawl the Site Like Googlebot

Log data tells you what Google is actually doing. A full site crawl tells you what's structurally available for it to find.

Run a complete crawl using a tool like Screaming Frog or Sitebulb, configured to behave like Googlebot (respecting robots.txt, following redirects, rendering JavaScript if your site relies on it). This crawl should surface:

• Orphan pages, meaning pages with no internal links pointing to them. These are nearly invisible to Googlebot regardless of how good the content is.

• Redirect chains, where a URL redirects to another redirect before reaching its final destination. Each hop costs crawl budget, and chains longer than two or three hops are worth fixing outright.

• Internal link depth, showing how many clicks it takes to reach a given page from the homepage. Pages buried more than four or five clicks deep tend to get crawled less often.

• Duplicate and thin content, which Google may crawl repeatedly without ever indexing.

Once you have this crawl data, compare it against the log data from Step 2. Pages your crawl found but your logs show Googlebot rarely visiting are strong candidates for either improved internal linking or reduced priority, depending on their actual value. Fixing internal linking is one of the highest-leverage moves here.

Step 4: Cross-Reference Sitemap, Crawl Data, and Index Status

This step turns raw data into a diagnosis. On its own, an XML sitemap doesn't tell you much. Compared against your actual index status and crawl data, it becomes a map of exactly where the problems are.

Split your sitemap by site section if it isn't already (for example, separate sitemaps for products, categories, and blog content). For each section, calculate:

Indexation ratio = (indexed URLs in that section) ÷ (total URLs submitted in that section's sitemap)

A section with a low ratio compared to the rest of the site tells you where to focus. If your blog content indexes at 90% but your product pages index at 40%, that's not a general crawl budget problem, it's a specific issue with how product pages are structured, linked, or valued by Google.

Layer in the log data from Step 2: is the low-indexing section also getting little crawl attention, or is Google crawling it plenty but declining to index it? The first points to a discovery or prioritization problem. The second points to a content quality or duplication problem, which a crawl budget fix alone won't solve that's a content quality problem, not a crawl budget one.

Common Crawl Budget Wasters on Large Sites

These are the issues that show up repeatedly across large, complex sites. Not every site has all of them, but most crawl budget problems trace back to one or more of the following.

• Faceted navigation and URL parameters. Filters, sorting options, and tracking parameters can generate thousands of near-duplicate URLs from a single set of products or listings a pattern we see constantly on Shopify stores.

• Orphan pages. Content with no internal links pointing to it, often left behind after a site restructure or migration.

• Redirect chains and loops. Old URLs redirecting through multiple hops, sometimes going back to where they started.

• Duplicate or thin content without canonical tags. Pages that overlap heavily in content but don't clearly signal which version should be indexed.

• Soft 404s and expired content left live. Out-of-stock products or expired listings that return a 200 status but have no real content, which Google keeps crawling without indexing.

• JavaScript-rendered content. Googlebot has to fetch the HTML and then render the JavaScript separately, which uses more crawl resources and happens on a delayed schedule we've written about whether JS-heavy sites are invisible to crawlers in more depth.

• Slow server response times or frequent 5xx errors. This directly reduces crawl capacity since Google throttles crawl rate to avoid overloading a struggling server.

For ecommerce and marketplace sites, faceted navigation is usually the single biggest offender this is one of the most common issues on ecommerce builds.

Prioritizing What to Fix First

Once you've identified the issues, the temptation is to fix everything at once. On a large site, that's rarely realistic, and it's usually unnecessary.

Prioritize based on two things: how much crawl budget an issue is consuming (from your log analysis) and how much it affects your highest-value page templates. A redirect chain affecting 200 low-traffic blog tags matters less than faceted navigation URLs eating crawl budget that should be going to your product pages.

A simple way to rank issues:

• High crawl waste, high-value pages affected. Fix these first. This is usually where faceted navigation or parameter issues touch your money pages directly.

• High crawl waste, low-value pages affected. Fix these next, often through blocking or consolidation rather than page-by-page cleanup.

• Low crawl waste, high-value pages affected. Worth fixing for other reasons (user experience, conversion), but not urgent from a crawl budget standpoint.

• Low crawl waste, low-value pages affected. Deprioritize. Fixing these won't move the needle.

Fixing the Issues: Quick Reference

Monitoring Crawl Budget After the Audit

A crawl budget audit isn't something you do once and forget. Sites that generate large amounts of new content or URLs on an ongoing basis need a recurring check, not just a one-time cleanup.

At minimum, revisit the Crawl Stats and Index Coverage reports monthly to catch trend changes early, particularly rising error rates or a growing "Discovered, not indexed" count. For sites with frequent structural changes (new product lines, site migrations, template changes), a lighter version of the log analysis from Step 2 on a quarterly basis will catch new waste before it compounds. Ongoing monitoring is what keeps this from becoming a recurring fire drill.

The goal isn't perfection. It's catching drift before it turns into a six-month indexing backlog.