Upcoming deprecation of Crawl Rate Limiter Tool in Search Console
Stay organized with collections
Save and categorize content based on your preferences.
Friday, November 24, 2023
The crawl rate limiter tool in Search Console is being deprecated on Jan 8th, 2024. This tool has
been
available for over a decade,
but with the improvements we've made to our crawling logic and other tools available to
publishers, its usefulness has dissipated.
Googlebot reacts to how the site–or more specifically the server handling the site–
responds to Googlebot's HTTP requests. For example, if the server persistently returns HTTP
500 status codes for a range of URLs, Googlebot will automatically, and almost
immediately slow down crawling. Similarly, Googlebot slows down automatically if the response time
for requests gets significantly longer. If you do experience unusually heavy crawling that your
site can't manage on its own, refer to this
help article.
In contrast, the rate limiter tool had a much slower effect; in fact it may have taken over a day
for the new limits to be applied on crawling. Fortunately though, site owners rarely had to resort
to using the tool, and those who have, in many cases set the crawling speed to the bare minimum.
With the deprecation of the crawl limiter tool, we're also setting the minimum crawling speed to a
lower rate, comparable to the old crawl rate limits. This means that we effectively continue
honoring the settings that some site owners have set in the past if the Search interest is low,
and our crawlers don't waste the site's bandwidth.
Because of the advances in the automated crawl rate handling, and in the spirit of keeping things
simple for users, we'll be deprecating this tool in Search Console. We are keeping the Googlebot
report form for reporting unusual
Googlebot activities and for emergency cases, but keep in mind that the fastest way to reduce
crawl rate is to instruct Googlebot through server responses as detailed in
our documentation.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eThe crawl rate limiter tool in Google Search Console will be deprecated on January 8, 2024, due to advancements in Google's crawling technology.\u003c/p\u003e\n"],["\u003cp\u003eGooglebot automatically adjusts its crawl rate based on website and server responses, like HTTP errors or slow response times.\u003c/p\u003e\n"],["\u003cp\u003eFor persistent crawling issues, refer to the provided help article for guidance on reducing Googlebot's crawl rate.\u003c/p\u003e\n"],["\u003cp\u003eSite owners experiencing unusually high crawl rates are encouraged to use the Googlebot report form for assistance.\u003c/p\u003e\n"],["\u003cp\u003eWhile the crawl rate limiter is being removed, Google will still honor previously set limits for sites with low search interest.\u003c/p\u003e\n"]]],["The Search Console's crawl rate limiter tool will be removed on January 8th, 2024. Googlebot now automatically adjusts crawling speed based on server responses, slowing down with HTTP `500` errors or increased response times. The tool's impact was slow, and it was rarely used. The minimum crawl speed will be reduced to match previous limits. While the tool is deprecated, reporting unusual Googlebot activity remains available via a report form, but the fastest way to adjust crawl rate is through server responses.\n"],null,["# Upcoming deprecation of Crawl Rate Limiter Tool in Search Console\n\nFriday, November 24, 2023\n\n\nThe crawl rate limiter tool in Search Console is being deprecated on Jan 8th, 2024. This tool has\nbeen\n[available for over a decade](/search/blog/2008/12/more-control-of-googlebots-crawl-rate),\nbut with the improvements we've made to our crawling logic and other tools available to\npublishers, its usefulness has dissipated.\n\n\nGooglebot reacts to how the site--or more specifically the server handling the site--\nresponds to Googlebot's HTTP requests. For example, if the server persistently returns HTTP\n`500` status codes for a range of URLs, Googlebot will automatically, and almost\nimmediately slow down crawling. Similarly, Googlebot slows down automatically if the response time\nfor requests gets significantly longer. If you do experience unusually heavy crawling that your\nsite can't manage on its own, refer to this\n[help article](/search/docs/crawling-indexing/reduce-crawl-rate).\n\n\nIn contrast, the rate limiter tool had a much slower effect; in fact it may have taken over a day\nfor the new limits to be applied on crawling. Fortunately though, site owners rarely had to resort\nto using the tool, and those who have, in many cases set the crawling speed to the bare minimum.\nWith the deprecation of the crawl limiter tool, we're also setting the minimum crawling speed to a\nlower rate, comparable to the old crawl rate limits. This means that we effectively continue\nhonoring the settings that some site owners have set in the past if the Search interest is low,\nand our crawlers don't waste the site's bandwidth.\n\n\nBecause of the advances in the automated crawl rate handling, and in the spirit of keeping things\nsimple for users, we'll be deprecating this tool in Search Console. We are keeping the Googlebot\n[report form](/search/docs/crawling-indexing/reduce-crawl-rate) for reporting unusual\nGooglebot activities and for emergency cases, but keep in mind that the fastest way to reduce\ncrawl rate is to instruct Googlebot through server responses as detailed in\n[our documentation](/search/docs/crawling-indexing/reduce-crawl-rate).\n\n\nIf you have questions or comments, write in our\n[Google Search Central Community](https://support.google.com/webmasters/community).\n\nPosted by [Gary Illyes](/search/blog/authors/gary-illyes) and [Nir Kalush](/search/blog/authors/nir-kalush)"]]
