# List of Google's special-case crawlers The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process. For example, `AdsBot` ignores the global robots.txt user agent (`*`) with the ad publisher's permission. The general [technical properties](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers#crawl-technical-props) of Google's crawlers also apply to the special-case crawlers. The special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers. The IP ranges are published in the [special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json) object. The special-case crawlers' reverse DNS mask matches `rate-limited-proxy-***-***-***-***.google.com`. The following list shows the special-case crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the `User-agent:` line in robots.txt, and the products that are affected by crawl preferences for the crawler. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about. > [!CAUTION] > **Caution** : The user agent string can be spoofed. [Learn how to verify if a requestor is from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests). ## [APIs-Google](https://developers.google.com/crawling/docs/crawlers-fetchers/apis-user-agent) : **User-Agent in HTTP requests** ``` APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) ``` **robots.txt** User-agent token in robots.txt `APIs-Google` *** ** * ** *** The global user agent (`*`) is ignored. Example robots.txt group ``` user-agent: APIs-Google allow: /archive/1Q84 disallow: /archive/ ``` **Affected products** Crawling preferences addressed to the `APIs-Google` user agent affect the delivery of push notification messages by Google APIs. ## AdsBot Mobile Web : **User-Agent in HTTP requests** ``` Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) ``` **robots.txt** User-agent token in robots.txt `AdsBot-Google-Mobile` *** ** * ** *** The global user agent (`*`) is ignored. Example robots.txt group ``` user-agent: AdsBot-Google-Mobile allow: /archive/1Q84 disallow: /archive/ ``` **Affected products** Crawling preferences addressed to the `AdsBot-Google-Mobile` user agent affect Google Ads' ability to check [web page ad quality](https://support.google.com/google-ads/answer/2404197). ## AdsBot : **User-Agent in HTTP requests** ``` AdsBot-Google (+http://www.google.com/adsbot.html) ``` **robots.txt** User-agent token in robots.txt `AdsBot-Google` *** ** * ** *** The global user agent (`*`) is ignored. Example robots.txt group ``` user-agent: AdsBot-Google allow: /archive/1Q84 disallow: /archive/ ``` **Affected products** Crawling preferences addressed to the `AdsBot-Google` user agent affect Google Ads' ability to check [web page ad quality](https://support.google.com/google-ads/answer/2404197). ## Google-Safety : |---|---| | **User-Agent in HTTP requests** | ``` Google-Safety ``` | | **robots.txt** | The Google-Safety user agent ignores robots.txt rules. | | **Affected products** | The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences. | ## `Mediapartners-Google` : **User-Agent in HTTP requests** Desktop agent ``` Mediapartners-Google ``` Mobile agent ``` (Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) ``` **robots.txt** User-agent token in robots.txt `Mediapartners-Google` *** ** * ** *** The global user agent (`*`) is ignored. Example robots.txt group ``` user-agent: Mediapartners-Google allow: /archive/1Q84 disallow: /archive/ ``` **Affected products** The `Mediapartners-Google` crawler visits participating sites in order to provide them with relevant ads. Crawling preferences addressed to the `Mediapartners-Google` user agent affect various ad-related Google products (for example, [Google AdSense](https://support.google.com/adsense/answer/99376) and [Google Ad Manager](https://support.google.com/admanager/answer/6023726)). ## Retired special-case crawlers The following special-case crawlers are no longer in use, and are only noted here for historical reference. ### AdsBot Mobile Web : **User-Agent in HTTP requests** ``` Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) ``` **robots.txt** User-agent token in robots.txt `AdsBot-Google-Mobile` *** ** * ** *** The global user agent (`*`) is ignored. **Affected products** Crawling preferences addressed to the `AdsBot-Google-Mobile` user agent affected Google Ads' ability to check iPhone [web page ad quality](https://support.google.com/google-ads/answer/2404197). ### Duplex on the web : **User-Agent in HTTP requests** ``` Mozilla/5.0 (Linux; Android 11; Pixel 2; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Mobile Safari/537.36 ``` **robots.txt** User-agent token in robots.txt `DuplexWeb-Google` *** ** * ** *** Duplex on the web may ignore the `*` wildcard. **Affected products** Supported the Duplex on the web service. ### [Google Favicon](https://developers.google.com/search/docs/appearance/favicon-in-search#crawler) : **User-Agent in HTTP requests** ``` Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon ``` **robots.txt** User-agent token in robots.txt `Googlebot-Image` *** ** * ** *** `Googlebot` ### Mobile Apps Android : **User-Agent in HTTP requests** ``` AdsBot-Google-Mobile-Apps ``` **robots.txt** User-agent token in robots.txt `AdsBot-Google-Mobile-Apps` *** ** * ** *** The `AdsBot-Google-Mobile-Apps` user agent obeyed [`AdsBot-Google`](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#adsbot) robots rules, but ignored the global user agent (`*`). **Affected products** Crawling preferences addressed to the `AdsBot-Google-Mobile-Apps` user agent affect Google Ads' ability to check Android app page [ad quality](https://support.google.com/google-ads/answer/2404197). ### Web Light : **User-Agent in HTTP requests** ``` Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19 ``` **robots.txt** User-agent token in robots.txt `googleweblight` *** ** * ** *** The `googleweblight` user agent was used only for explicit browse requests of a human visitor, and so it ignored robots.txt rules, which are used to block automated crawling requests. **Affected products** The Web Light user agent checked for the presence of the `no-transform` header whenever a user clicked your page in Search under appropriate conditions.