Crawl Budget: What It Is and How to Optimize It

Crawl budget is the number of pages Googlebot is willing and able to crawl on your site within a given timeframe. It’s determined by two factors: crawl capacity (how much load your server can handle without slowing down) and crawl demand (how much Google actually wants to crawl your site, based on popularity and how often content changes).

For most small to mid-sized sites, crawl budget isn’t a real constraint — Google can typically crawl everything they need to. It becomes genuinely relevant for large sites (100,000+ pages), sites with frequently changing content (news, e-commerce with dynamic inventory), or sites that generate large numbers of low-value URLs (faceted navigation, session IDs, infinite filter combinations).

How crawl budget gets wasted: - Duplicate URLs from parameters: Filter and sort options that create dozens of near-identical URL variants for the same content. - Soft 404s: Pages that return a 200 status but show “no results” or empty content, which Google still has to crawl to figure out they’re low value. - Redirect chains: Each hop consumes crawl budget without adding indexable value. - Slow server response times: A slow-loading server directly reduces how many pages Googlebot can crawl in a session.

How to optimize it: Block low-value URL parameters in robots.txt, fix or remove soft 404s, consolidate duplicate content with canonical tags, and improve server response times so each crawl session covers more ground.

Does your site actually have a crawl budget problem?

Before spending time on crawl budget, check whether it's really a constraint. Signs that it is:

• New or updated pages take weeks to be crawled, even though they are in the sitemap and linked internally.
• The Page indexing report shows a large number of URLs as "Discovered – currently not indexed".
• The site has far more crawlable URLs than real pages, often because of filters, parameters or calendars.

If your site has a few thousand pages and new content is indexed within days, crawl budget is not your bottleneck. Focus on content quality and internal linking instead.

How to read the Crawl Stats report

In Search Console, go to Settings → Crawl stats. It shows total crawl requests, total download size and average response time over the last 90 days. Look at:

• By response: a high share of 404, 301 or 5xx responses means Googlebot is wasting requests.
• By file type: unexpected volumes of JSON or JavaScript requests can point to crawlable API endpoints.
• By purpose: "Discovery" vs "Refresh" shows whether Google is finding new URLs or rechecking known ones.
• Host status: any server availability problems here reduce how much Google is willing to crawl.

For the full picture, compare this with your server logs. Our log file analysis guide explains how.

Handling URL parameters today

Google retired the URL Parameters tool in Search Console in 2022, so parameter handling now has to happen on your site:

• Block parameters that never create useful pages (session IDs, sort orders, internal tracking) in robots.txt, for example Disallow: /*?sort=.
• Add canonical tags from filtered URLs to the main category where the filtered page has no search demand.
• Don't link internally to parameter URLs unless you want them crawled.
• For faceted navigation, decide which filter combinations deserve indexable pages and keep the rest out of the crawl.

Help Google crawl your important pages first

• Keep the XML sitemap clean: only canonical, indexable URLs that return 200, with accurate lastmod dates.
• Link to new and updated pages from strong, frequently crawled pages such as the homepage or category hubs.
• Return a real 404 or 410 for removed pages instead of redirecting everything to the homepage.
• Fix redirect chains so every redirect goes straight to the final URL.

Server speed and crawl rate

Google automatically crawls more when your server responds quickly and slows down when it sees errors or slow responses. Keeping response times low with caching and a good host is one of the most direct ways to increase crawl capacity. If Googlebot is overloading your server, return 503 or 429 status codes temporarily. Google will slow down, but long periods of these errors will reduce crawling and can drop pages from the index.