APIs-Google User Agent

Google's user agent for API push notifications respects APIs-Google (as opposed to Googlebot) robots.txt directives.

Introduction

If you administer a website that allows developers to access Google's APIs, you might see requests in your access logs that specify a user-agent of APIs-Google:

User-Agent: APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)

APIs-Google is the user agent used by Google APIs to deliver push notification messages. Application developers can request these notifications to avoid the need for continually polling Google's servers to find out if the resources they are interested in have changed. To make sure nobody abuses this service, Google requires developers to prove that they own the domain before allowing them to register a URL with a domain as the location where they want to receive messages.

Push notifications are supported by only a subset of Google APIs, including Drive and Cloud Storage.

For webmasters: APIs-Google and your site

How APIs-Google accesses your site

APIs-Google sends each push notification using an HTTPS POST request. If the request fails due to an error condition that might be temporary, APIs-Google will try sending the notification again. If the request still doesn't succeed, it will continue to retry—based on an exponential backoff schedule—up to a maximum of several days.

For example, if the application fails to acknowledge each push notification request promptly with one of the success HTTP status codes (such as 200), APIs-Google might consider this a timeout error and start retrying the notification. There are also a number of retryable error HTTP status codes; for example 503 is used when the application is temporarily down or overloaded.

The rate at which APIs-Google accesses your site varies by how many push notification requests were created for servers on your site, by how fast the monitored resources are getting updated, and by the number of retries occurring. As a result, the APIs-Google traffic patterns can be consistent in some scenarios, but in other scenarios the traffic can be sporadic or spiky.

Blocking APIs-Google from calling into your site

If you administer a domain that has subdomains or URL subspaces that are owned or administered separately, one of those other owners may have set up applications that use push notifications. If you want to block APIs-Google, we suggest that you first attempt to contact anyone who might have set up an application like this. Alternatively, you can use the standard robots.txt directives to block APIs-Google from accessing your site. The user-agent to specify in the robots.txt file is APIs-Google. Since APIs-Google does not follow Googlebot directives, it can be controlled separately from other Google crawlers.

If you decide to create a robots.txt file, there may be a small delay before APIs-Google discovers your changes. If APIs-Google continues to send messages to your site several days after you've blocked it in robots.txt, check that the robots.txt is in the correct location. It must be in the top directory of the server, for example, www.example.com/robots.txt; placing the file in a subdirectory won't have any effect.

Making sure your site is APIs-Google friendly

Since APIs-Google uses HTTPS to deliver push notifications, it requires your site to have a valid SSL certificate. Invalid certificates include:

  • Self-signed certificates.
  • Certificates signed by an untrusted source.
  • Certificates that have been revoked.

Also, to avoid unnecessary retry requests, the applications must be well-designed and respond promptly (within seconds) to notification messages.

Handling potentially fake requests

The IP addresses used by APIs-Google change from time to time. In addition, anyone can set the user-agent to whatever they like. The best way to confirm that such accesses are from Google is to use a reverse DNS lookup that's similar to the kind you would use to verify that a bot accessing your server really is Googlebot. In this case, search in your logs for any IP addresses identified with the APIs-Google user agent. The reverse DNS lookup should identify a "googlebot.com" domain.