What shapes it
robots.txt decides which URLs may be requested. Server behaviour decides how fast: Google’s documentation says that 5xx and 429 responses make its crawlers slow down, that the crawl rate falls in proportion to the number of URLs returning errors and that it recovers gradually after the server returns 2xx again. Google’s crawlers follow up to 10 redirect hops by default. Persistent server errors can lead to URLs being dropped from the index.
Crawl budget
Large sites have a limit to how much a search engine will fetch in a given time. Parameter URLs, redirect chains and soft errors waste it. For small client sites it rarely matters, while availability does.
Where to look
In Google Search Console the Crawl stats report shows requests per day, average response time and the host status for robots.txt, DNS and server connectivity. A rise in 5xx answers or in response time on a client site often comes before a fall in crawling. A quick manual test is curl -I https://example.com/robots.txt and the same for one important page: both should answer 200 quickly. A robots.txt that fails with a 5xx error, even briefly, can pause crawling of the whole site.
What to check
That important pages return 200 reliably, that redirects are short and that nothing blocks the files a page needs.
Sources
Read on . Standards and vendor pages change; confirm the current text before you rely on it.
- Google Search Central, How HTTP status codes, network and DNS errors affect Google Search (developers.google.com)