Google has simply pushed some updates to the cut back the Google crawl charge documentation. A lot of the adjustments are simply organizational to the content material on the web page however beneath the “Urgently cut back crawler site visitors (for emergencies)” part, Google added new info on how you can use Retry-After HTTP header.
Google stated it has “Restructured the emergency crawl charge discount part and added info and examples concerning the Retry-After HTTP header to the Scale back the Google crawl charge documentation.” Google added it did this to “Help for the Retry-After HTTP header isn’t new (it was already documented in Briefly pause or disable a web site), and including it on to the crawl charge discount information makes it simpler to search out and perceive when urgently decreasing crawler site visitors.”
In case you examine the reside doc to the archived doc, you’ll largely see a variety of issues moved across the web page.
The most important addition is the part on:
When returning a 503 or 429 standing code, you may also embrace a Retry-After HTTP header (as outlined in RFC 9110 HTTP Semantics) to point when Google’s crawlers can retry the request, utilizing both a delay in seconds or an absolute UTC date and time:
Google added instance code as effectively, here’s a screenshot:
Google is not introducing a brand new method to throttle Googlebot, the underlying mechanism hasn’t modified. Google already used 500, 503, and 429 responses as alerts to scale back crawling. The brand new documentation makes Retry-After an express extra sign that web site homeowners can present when utilizing 503 or 429.
Discussion board dialogue at X.
