What is crawl budget and when to pay attention to it - Zephyra Studio

Crawl budget is the number of pages Googlebot can and wants to review on your site over a period, for example in a single day. Google decides on its own how much resource to spend on your domain, based on its size, speed, and how often the content changes. That sounds abstract, but it has real consequences: if Googlebot spends that budget on thousands of pointless URLs, new and important pages wait longer to be crawled. For a site with fifty pages crawl budget is not a topic. For a catalogue with tens of thousands of products it is one of the main technical problems.

How Google decides how much to crawl

Google does not crawl a site evenly. Each domain gets an approximate ceiling for how many pages it can visit per unit of time, and that number changes. New sites and sites that go down often get less; older, fast, regularly updated sites get more.

Three things shape that estimate: how big the site is and how often it changes, how fast the server responds, and how much of what Googlebot finds is genuinely new or useful.

When crawl budget becomes a real problem

For smaller sites, crawl budget is theory. Google visits everything, quickly. The problem starts when the number of URLs explodes while most of them lead nowhere useful.

A classic example is an online store with filters. One category with five filters can produce hundreds of URL combinations, each showing the same or nearly the same content. Googlebot spends time on those combinations while a real product or a new category waits in the queue.

What burns budget for nothing

Pointless URLs burn budget: sorting, filters, tracking parameters, date calendars, searches with no results. The same applies to pages that return an error but are still linked, and to endless sequences of self-generated pages.

Google Search Console has a Crawl stats report showing how many requests Googlebot sent and where. If parameter URLs dominate there, that is a clear sign of where your budget goes.

A practical fix: help Googlebot out

First rule: do not let Googlebot into what should not be indexed. You can block filters and internal search through robots.txt or set noindex, depending on whether you want the URL to appear in the index.

Second: internal links should point at the pages that matter. If every page links to hundreds of others, Googlebot goes everywhere, including where it should not. An XML sitemap with only the important URLs helps make the priorities clear.

Related terms

For the wider picture, see also:

Source

Key takeaways

• Crawl budget is how many pages Googlebot reviews on your site in a given period.
• For a site of a few hundred pages it is practically irrelevant; for a large catalogue it is not.
• Filters, sorting, parameters, and empty searches are the most common waste of budget.
• The Crawl stats report in Search Console shows where Googlebot actually spends its time.

Conclusion

Crawl budget is not solved with a single move, but with ongoing cleanup of what Googlebot has no reason to visit. Our technical SEO work starts with a review of Crawl stats and URL structure, because without that, even the best content does not reach the index in time.