Changelog

This page details the latest major updates made to Google's crawling documentation.

To get the latest crawling documentation updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://developers.google.com/crawling/docs/changelog/crawling_docs_updates.rss.

November 2025

November 20

Migrated documentation to the new Google crawling documentation site

What: Migrated crawling documentation from the Google Search Central to a new Google crawling documentation site. The content hasn't changed, only the location. Notable moves include:

• Overview of Google crawlers and fetchers
• Verify requests from Google crawlers and fetchers
• Reduce the Google crawl rate
• Google common crawlers
• How Google interprets the robots.txt specification

Why: Google's crawling infrastructure is shared across a variety of Google products beyond Search, including Google Shopping, News, Gemini, AdSense, and more. The new site is a more logical home for this documentation and makes it easier to document new features and updates that are relevant to all of these products.

November 12

Added the Google-Pinpoint fetcher

What: We added the Google-Pinpoint fetcher to the list of user-triggered fetchers.

Why: The Google-Pinpoint fetcher is used by the Pinpoint research tool.

November 3

Added a new user-triggered fetcher

What: Based on feedback, we added the Google-CWS fetcher to the list of user-triggered fetchers.

October 2025

October 15

Updated the list of Google products that use the Read Aloud service

What: Update the documentation on Google Read Aloud with an updated list of Google products that use the Read Aloud service.

Why: Other Google products can now use the Google Read Aloud service.

October 9

Added Google-NotebookLM to the list of user-triggered fetchers

What: Based on feedback, we added Google-NotebookLM to the list of user-triggered fetchers.

July 2025

July 1

Updated the Google Read Aloud user agent

What: Updated Google Read Aloud user agent in HTTP requests with newer browser versions.

Why: To accommodate sites which don't support old browser versions.

April 2025

April 25

Updated the description of the Google-Extended product token

What: Based on publisher feedback, we updated the Google-Extended product token description to provide additional specificity and clarity.

Corrected the description of the crawler preferences addressed to the Googlebot-News user agent

What: Updated the Googlebot-News user agent description.

Why: The description for how crawling preferences addressed to Googlebot-News mistakenly stated that they'd affect the News tab on Google, which is not the case.