Webmasters

Appendix: Google's website crawlers

This table covers the common Google crawlers, the user-agent they follow for the robots exclusion protocol as well as the user-agent specified in HTTP(S) requests. Google also uses some other user-agents, not listed here, to fetch content in real time in response to a user's action.

Crawler User-agents (+ less-specific alternatives) followed in robots.txt, robots meta tags & X-Robots-Tag User-agent in HTTP(S) requests Comments + Documentation URL
Googlebot (web) Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Alternate (rarely used):
Googlebot/2.1 (+http://www.google.com/bot.html)
Generic Googlebot crawler for web-search
http://www.google.com/bot.html
Googlebot News Googlebot-News
(Googlebot)
Googlebot-News For Google News
Googlebot Images Googlebot-Image
(Googlebot)
Googlebot-Image/1.0 For Image Search
Googlebot Video Googlebot-Video
(Googlebot)
Googlebot-Video/1.0 For Video Search
Googlebot Mobile Googlebot-Mobile
(Googlebot)
[various mobile device types] (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) For Google Mobile web-search results
http://www.google.com/support/mobile/bin/answer.py?answer=37425
Google Mobile AdSense Mediapartners-Google
Mediapartners
(Googlebot)
[various mobile device types] (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)  
Google AdSense Mediapartners-Google
Mediapartners
(Googlebot)
Mediapartners-Google  
Google AdsBot landing page quality check AdsBot-Google
(Given the special nature of this crawler, only directives for this user-agent are followed.)
AdsBot-Google (+http://www.google.com/adsbot.html) Only visits landing pages used in AdWords campaigns. See http://www.google.com/adsbot.html

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.