Googlebot crawl rate tool in Search Console is going away

Google is discontinuing the crawl rate limiter tool in Search Console effective January 8, 2024, citing its lack of usefulness. This feature, a legacy tool within Google Search Console, allowed users to regulate Google’s site crawl frequency. Google mentioned advancements in their crawling logic and available publisher tools as reasons for its removal.

The crawl rate limiter, housed in the legacy version of Google Search Console, permitted users to regulate Google’s crawling frequency, typically advised against unless server overload issues arose due to Googlebot activity. Until its removal, users can access the tool.

Google’s decision to remove this tool stems from their improved crawling logic and other available publisher tools, rendering the limiter’s utility obsolete.

Gary Illyes from Google clarified that Googlebot’s crawling pace is reactive to a site’s server response. If a server returns persistent HTTP 500 status codes or encounters significantly longer response times, Googlebot automatically adjusts its crawling pace. The rate limiter tool in Google Search Console had a slower effect and was scarcely used, typically resulting in minimum crawling speed settings.

With the tool’s deprecation, Google aims to maintain a lower minimum crawling speed, similar to the old crawl rate limits, for sites with low search interest to prevent bandwidth wastage.

For users accustomed to the crawl rate tool, be aware of its impending removal and assess any potential impact on your server performance once it’s phased out.