Click here to see your recently viewed pages and most viewed pages.

Appendix: Google's website crawlers

This table covers the common Google crawlers, the user-agent they follow for the robots exclusion protocol as well as the user-agent specified in HTTP(S) requests. Google also uses some other user-agents, not listed here, to fetch content in real time in response to a user's action.

Crawler User-agents (+ less-specific alternatives) followed in robots.txt, robots meta tags & X-Robots-Tag User-agent in HTTP(S) requests Comments + Documentation URL
Googlebot (web) Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +
Alternate (rarely used):
Googlebot/2.1 (+
Generic Googlebot crawler for web-search
Googlebot News Googlebot-News
Googlebot-News For Google News
Googlebot Images Googlebot-Image
Googlebot-Image/1.0 For Image Search
Googlebot Video Googlebot-Video
Googlebot-Video/1.0 For Video Search
Googlebot Mobile Googlebot-Mobile
[various mobile device types] (compatible; Googlebot-Mobile/2.1; + For Google Mobile web-search results
Google Mobile AdSense Mediapartners-Google
[various mobile device types] (compatible; Mediapartners-Google/2.1; +  
Google AdSense Mediapartners-Google
Google AdsBot landing page quality check AdsBot-Google
(Given the special nature of this crawler, only directives for this user-agent are followed.)
AdsBot-Google (+ Only visits landing pages used in AdWords campaigns. See