What is crawl budget, and when does it matter?

Google says its crawl budget guide is an advanced guide, written mainly for:

• Large sites with 1 million or more unique pages whose content changes moderately often, about once a week.
• Medium or larger sites with 10,000 or more unique pages whose content changes very rapidly, about daily.
• Sites with a large share of URLs marked "Discovered, currently not indexed" in Search Console.

Google adds that these numbers are rough estimates, not exact thresholds. If your site has a few hundred pages, crawl budget is very unlikely to be your problem.

Google wants to crawl your site without overwhelming your server. The crawl capacity limit caps how much crawling your server is asked to handle at once.

Google says every site starts with the same conservative default. The limit can rise if your site responds quickly for a while, and it falls if your site slows down or returns server errors.

Crawl demand is how much Google wants to crawl. Google bases it on how many URLs it knows about, how popular they are and how often they change. Google notes that if many known URLs are duplicates, or pages you do not want crawled, that wastes crawling time. Site-wide events such as site moves can also raise crawl demand for a while.

From Google's guide, the practices that make the biggest difference:

1. Cut duplicate URLs. Parameters, filters and session IDs can create thousands of addresses for the same content.
2. Keep your sitemap up to date, so Google finds the URLs that matter.
3. Return the right status codes. Pages that are gone should say so, instead of returning a page that looks normal.
4. Make pages fast. A server that responds quickly can be crawled more.
5. Block what Google should never crawl with robots.txt.

One warning Google gives directly: do not use robots.txt to temporarily reallocate crawl budget to other pages. Google says it will not shift that freed budget elsewhere unless it is already hitting your site's crawl capacity limit.

Search Console's Crawl Stats report shows how often Google requests your pages, how fast your server responds and which response codes it returns. Google's overview of its crawlers explains which Googlebot types visit and how to recognize them in your server logs.

Our read: for a typical small or mid-size business site, "Google is not crawling my page" is almost never a budget problem. More often the page is not linked from anywhere, is blocked by mistake, or looks too similar to another page. Fix those first.

Is crawl budget a ranking factor? Crawling decides whether Google can see a page at all. Being crawled more often does not by itself make a page rank higher.

Can I ask Google to crawl more? You can make crawling easier: faster responses, fewer wasted URLs, an accurate sitemap.

Does a sitemap guarantee crawling? No. It helps Google discover URLs, but crawling still depends on capacity and demand.

Crawl budget is real but niche. If your site is not huge or fast-changing, spend your effort on clean URLs, internal links and helpful pages instead.