Google Compute Engine

Authenticating from Google Compute Engine

You can authenticate from Google Compute Engine to other Google services using OAuth 2.0 authentication. Depending on your use case, you can either perform the standard OAuth 2.0 flow or use service accounts, as described here.

Not what you're looking for? Are you looking for documentation on how to authorize requests to Google Compute Engine?

Contents

Overview

When you write applications that need to talk to other Google services, you need to authenticate your application to these services before you can perform any operations or access any data. Often times, you can authenticate these applications using the standard OAuth 2.0 flow, but Google Compute Engine makes this process simpler by providing service accounts.

When you create a Google Compute Engine project, Google Compute Engine also creates a service account for that project. This service account authenticates the project to other Google services but you can also use the service account to authenticate your instances to other APIs. This is ideal for cases when you have applications running within instances that need to programmatically talk between services but don't need access to user data.

For example, lets say you want to write an application that runs within a VM instance. This application saves and retrieves log files to and from Google Cloud Storage but in order for the application to talk to Google Cloud Storage, it needs to authenticate to the Google Cloud Storage API. You can use the standard OAuth 2.0 flow (as briefly described in the table below) to do so or you can save yourself several steps if you set up your instance to use service accounts. Your application can then authenticate seamlessly to Google Cloud Storage and other Google APIs without having to perform the full OAuth 2.0 flow, which is handled by the instance. Here is a table the briefly outlines the steps required for each authentication process:

Standard OAuth 2.0 Flow Service Accounts
  1. Set up your application to construct an OAuth 2.0 authorization request to the Google OAuth 2.0 authorization server.
  2. Log in and consent to the access.
  3. The server returns an authorization code.
  4. Exchange your authorization code for a token.
  5. Use the token to make requests to the API.
  1. Create an instance with access to a service account.
  2. Set up your application to query the metadata server with a simple REST request to obtain a token.
  3. Use the token to make requests to the API.

If you don't need access to user information, using a service account is a quicker, easier way to set up your application to access other APIs. This document discusses:

  • Using service accounts with your applications

    If you want to run an application from within your instance that needs access other Google services, you can setup your application to use service accounts.

  • Using service accounts with tools and libraries

    When you set up a service account with your instance, it automatically allows tools and libraries within that instance, such as gsutil, to use the service account credentials to access the respective API, without requiring additional authentication on your end. See Using Service Accounts with Tools and Client Libraries for more information.

Note: Google Compute Engine service accounts are different than Developers Console service accounts. For more information on how they are different and how to use Developers Console service accounts instead of Google Compute Engine service accounts, see Developers Console service accounts.

Service Account Names

All Google Compute Engine projects can have one Google Compute Engine service account associated with them and multiple Developers Console service accounts. In the API, the Google Compute Engine service account is referred to as the default account, but you can find out the actual service account name by querying the metadata server. Generally, the account name has the format:

123845678986@project.gserviceaccount.com

Preparing an Instance to Use Service Accounts

Before you can use a service account for your application or tools, you must first set up your instance with a service account. To do so, provide the --service_account_scopes=<scope> flag during instance creation. The OAuth2 scope you provide determines the level of access your instance has to that service:

gcutil --project=<project-id> addinstance <instance name> --service_account_scope=<scope>

Behind the scenes, Google Compute Engine uses these scopes to request a refresh token that it holds on to. Although you cannot directly access the refresh token yourself, you can use the metadata server to request an access token that you can use in requests.

Note: A Google Compute Engine project can only hold a certain number of refresh tokens. If you reach this limit, you will need to free up refresh tokens before you can create additional instances with service account scopes. For more information, see Service Account Token Limits.

For example, you can create a new instance and authorize it to use a service account to access Google Cloud Storage with full control like so:

gcutil --project=<project-id> addinstance <instance name> \
  --service_account_scopes=https://www.googleapis.com/auth/devstorage.full_control

To specify multiple scopes, so your instance can access multiple services, provide a list of comma-separated scopes:

gcutil --project=<project-id> addinstance <instance name> \
  --service_account_scopes=https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/compute

Keep in mind the following as you're setting up your instance to use service accounts:

  • Once you have created an instance with a service account and specified scopes, you cannot change or expand the list of scopes.

    You should specify all desired scopes at instance creation time.

  • Each Google service has its own unique scopes and you'll need to find the right scope for your desired service.

    Generally, scopes are listed in the API documentation of the service. For example, see Google Cloud Storage Developer's Guide for more specific scope information.

  • Currently, it is only possible to authorize your instance for the service account of the project it belongs to.

    It is not possible to specify multiple service accounts or service accounts of other projects.

Service Account Scope Aliases

Usually, OAuth 2.0 scopes are long URIs that can be difficult to remember. For example, the Google Cloud Storage OAuth 2.0 scopes all begin with the URI:

https://www.googleapis.com/auth/devstorage.<scope>

For convenience, gcutil provides a list of aliases that you can use in place of the longer scope URIs. Here is the current list of available aliases:

Service Full Scope Alias Description
Google BigQuery https://www.googleapis.com/auth/bigquery bigquery Access to Google BigQuery API
Google Cloud SQL https://www.googleapis.com/auth/sqlservice cloudsql Access to Google Cloud SQL API
Google Compute Engine https://www.googleapis.com/auth/compute.readonly compute-ro Read-only access to Google Compute Engine
https://www.googleapis.com/auth/compute compute-rw Read-write access to Google Compute Engine
Google Cloud Storage https://www.googleapis.com/auth/devstorage.read_only storage-ro Read-only access to Google Cloud Storage
https://www.googleapis.com/auth/devstorage.read_write storage-rw Read-write access to Google Cloud Storage
https://www.googleapis.com/auth/devstorage.full_control storage-full Full access to Google Cloud Storage
Google App Engine Task Queue https://www.googleapis.com/auth/taskqueue taskqueue Access to Google App Engine Task Queue API

Specify the alias the same way you would specify the normal scope URI:

gcutil --project=<project-id> addinstance <instance name> \
  --service_account_scopes=storage-full

Note: These aliases are only recognized by gcutil and aren't recognized by the API or other libraries and tools. For those situations, you would need to specify the full scope URI.

Using Service Accounts with Applications

You can use service accounts with your applications to authenticate to other Google services. To do so:

  1. Make sure your instance has been set up to use a service account

    If not, follow the instructions provided above under Preparing an Instance to Use Service Accounts.

  2. Query the metadata server for an access token

    One you have created your instance, you need to query the metadata server for an access token. Here's how to do so using curl:

    curl "http://metadata/computeMetadata/v1/instance/service-accounts/default/token" -H "X-Google-Metadata-Request: True"

    Your token automatically has access to any of the scopes you specified when you initially started the instance. It is not possible to ask for a token outside of the scopes you originally indicated.

    The server returns something similar to the following:

    {
      "access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
      "expires_in":3599,
      "token_type":"Bearer"
    }
  3. Use the access token in requests to the desired API

    Use the access_token provided by the metadata server to perform requests to the API. Note that you can't use the access token for a scope that you didn't specify. For example, you can not use the access token above to access the Google Prediction API because the scope was only authorized for the Google Cloud Storage API and the Google Compute Engine API.

As a security feature, access tokens are designed to expire after a short period of time. Once an access token reaches 5 minutes of its expiration window, the metadata server no longer caches it. Within the 5 minute expiration window, you can request a new access token to use.

If you need a refresh token, which never expires and is used to request new access tokens, you need to perform the standard OAuth 2.0 flow.

Python Example

Here is a basic Python example that demonstrates how to request a token and access the Google Cloud Storage API from within an instance. This sample:

  1. Connects to the metadata server
  2. Provides a scope for access (in this case, the scope is for read-write access to Google Cloud Storage)
  3. Queries the metadata server
  4. Extracts the access token from the response
  5. Uses the access token to make a request to Google Cloud Storage

Note that we didn't need to construct an OAuth 2.0 authorization request to the Google OAuth 2.0 authorization server.

#!/usr/bin/python

import json
import urllib
import httplib2

METADATA_SERVER = 'http://metadata/computeMetadata/v1/instance/service-accounts'
SERVICE_ACCOUNT = 'default'
GOOGLE_STORAGE_PROJECT_NUMBER = 'YOUR_GOOGLE_STORAGE_PROJECT_NUMBER'

"""This script grabs a token from the metadata server, and queries the
Google Cloud Storage API to list buckets for the desired project."""

def main():
  token_uri = '%s/%s/token' % (METADATA_SERVER, SERVICE_ACCOUNT)
  http = httplib2.Http()
  resp, content = http.request(token_uri, method='GET', body=None, headers={'X-Google-Metadata-Request': 'True'}) # Make request to metadata server

  if resp.status == 200:
    d = json.loads(content)
    access_token = d['access_token'] # Save the access token

    # Construct the request to Google Cloud Storage
    resp, content = http.request('https://storage.googleapis.com', \
                                  body=None, \
                                  headers={'Authorization': 'OAuth ' + access_token, \
                                           'x-goog-api-version': '2', \
                                           'x-goog-project-id': GOOGLE_STORAGE_PROJECT_NUMBER })

    if resp.status == 200:
       print content
    else:
       print resp.status

  else:
    print resp.status

if __name__ == '__main__':
  main()

Using Service Accounts with Tools and Client Libraries

By default, tools that are provided in your instance, such as gsutil and gcutil, recognize and use a service account if it is available. For gsutil (and access to Google Cloud Storage), use one of the scopes defined at the Google Cloud Storage developer's guide and for gcutil, use one of the following scopes:

Scope Meaning
https://www.googleapis.com/auth/compute Read-write access to Google Compute Engine methods.
https://www.googleapis.com/auth/compute.readonly Read-only access to Google Compute Engine methods.

This only applies to tools that are provided automatically with the instance. If you created new tools, or added custom tools, you need to authorize your application as described in Using Service Accounts with your Application.

Checking if an Instance Uses Service Accounts

To find out if an instance is using service accounts, you can do either of the following:

  • Query the metadata server from within that instance

    To query the metadata server from within the instance itself, make a request to http://metadata/computeMetadata/v1/instance/service-accounts/. Here's how you would use curl to do so:

    user@myinst:~$ curl "http://metadata/computeMetadata/v1/instance/service-accounts/" -H "X-Google-Metadata-Request: True"
    761881434886@project.gserviceaccount.com/
    default/
  • Use gcutil from your local machine

    Run the following command from your local machine:

    gcutil getinstance <instance-name> --print_json --project=<project-id>

The response returned from either method should contain the following:

{
   "serviceAccounts":[
      {
         "email":"123845678986@project.gserviceaccount.com",
         "scopes":[
            "https://www.googleapis.com/auth/devstorage.full_control"
         ]
      }
   ]
}
  • If your instance is using a Google Compute Engine service account:

    The service account name will appear in the response. In this example, the service account 123845678986@project.gserviceaccount.com is associated with the Google Cloud Storage scope.

  • If your instance isn't using a Google Compute Engine service account:

    You will receive an empty serviceAccounts list.

Back to top

Service Account Token Limits

When you set up an instance to use a service account, Google Compute Engine automatically creates OAuth2 refresh tokens for your service account to use for authenticating to other Google services. Google Compute Engine creates OAuth2 refresh tokens for each VM instance that uses a unique set of {service_account, service_account_scopes} and reuses refresh tokens for each VM instance that uses an existing set of {service_account, service_account_scopes}. For example, if you have three instances:

Instance name service_account service_account_scopes
instance1 default scope1, scope2
instance2 default scope2, scope1
instance3 default scope1, scope2, scope3

Google Compute Engine creates two refresh tokens: one for the set {default, scope1, scope2} and one for the set {default, scope1, scope2, scope3}. Google Compute Engine does not create an additional refresh token for instance2 because service_account_scopes are unordered sets, so {scope1, scope2} is equivalent to {scope2, scope1} and can use the same refresh token.

There is a limit to the total number of refresh tokens that your service account can have at any one point in time. Currently, this limit is 600. If this limit is reached, Google Compute Engine will not be able to create an instance which requires a new refresh token, and you will get a SERVICE_ACCOUNT_TOO_MANY_TOKENS error. For example, if you have reached the refresh token limit, and you attempt to create an instance with a new, unique set {default, scope1, scope2, scope3, scope4), the action fails and you will receive the SERVICE_ACCOUNT_TOO_MANY_TOKENS error.

However, if Google Compute Engine already has a refresh token for an existing set, you can create new instances with that same set and it will reuse the same refresh token. For example, if you created an instance4 with the same set as instance1 {default, scope1, scope2}, the instance creation will succeed because you are not creating a new refresh token.

How to Free Up Refresh Tokens

If you reach the refresh token limit and cannot create new refresh tokens, you must delete some refresh tokens associated with the service account. To do so, delete Google Compute Engine instances that are using the service account, in order to reduce the number of distinct service_account_scopes sets used by the service account.

Note: If you add the “default” service account as a member to other projects, and it has created refresh tokens in those projects, you may have to track down and delete instances in those projects. In addition, it is possible that other Google Services may be using the service account, in which case you must track down those uses and free them.

Back to top

Google Developers Console Service Accounts

Currently, Google offers two different types of service accounts that you can use. We are working on clarifying the terminology here, but until then, here are how the two different types of service accounts differ:

  • Developers Console Service Accounts

    Developers Console service accounts lets you write applications which can authenticate themselves to access a compatible Google API using a JSON Web Token (JWT). The JWT can be used to acquire OAuth2 access tokens that can access Google APIs. You can write applications on Google Compute Engine that uses Developers Console service accounts, by following the Developers Console Service Accounts documentation.

  • Google Compute Engine Service Accounts

    Google Compute Engine service accounts work without the need to create or manage a JWT. Instead, programs running within your Google Compute Engine instances can automatically acquire OAuth2 access tokens with credentials to access any service in your Google API project (and any other services that granted access to that service account).

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.