Google Cloud Storage

Migrating from Amazon S3 to Google Cloud Storage

This page describes how to migrate from Amazon Simple Storage Service (Amazon S3) to Google Cloud Storage for users sending requests using an API. If you are not currently using Amazon S3 and you want to send requests using the Google Cloud Storage API, then start here: XML API Overview.

If you are new to Google Cloud Storage and will not be using the API directly, consider using the Google Developers Console. The Google Developers Console provides a graphical interface to Google Cloud Storage that enables you to accomplish many of your storage tasks using just a browser.

Overview of migration

If you are an Amazon S3 user, you can easily migrate your applications that use Amazon S3 to use Google Cloud Storage. You have two migration options:

Simple Migration

This is this easiest way to get started with Google Cloud Storage if you are coming from Amazon S3 because it requires just a few simple changes to the tools and libraries you currently use with Amazon S3. For more information, see Simple Migration.

While a simple migration allows you to get going quickly with Google Cloud Storage, it does not allow you to use all the features of Google Cloud Storage. To take full advantage of Google Cloud Storage, follow the steps for a full migration.

Full Migration
A full migration from Amazon S3 to Google Cloud Storage requires a few extra steps than a simple migration, but the benefit is that you can use all the features of Google Cloud Storage, including support for service accounts, multiple projects, and OAuth 2.0 for authentication. For more information, see Full Migration.

Simple migration

In a simple migration from Amazon S3 to Google Cloud Storage, you can use your existing tools and libraries for generating authenticated REST requests to Amazon S3, to also send authenticated requests to Google Cloud Storage. The changes you need to make to your existing tools and libraries are described in this section.

To get set up for a simple migration do the following:

  • Set a default Google project.
  • Get a developer key.
  • In your existing tools or libraries, make the following changes:
    • Change the request endpoint to use the Google Cloud Storage request endpoint.
    • Replace the Amazon Web Services (AWS) access and secret key with the corresponding Google Cloud Storage access key and secret key (collectively called your Google developer key).

That's it! At this point you can start using your existing tools and libraries to send keyed-hash message authentication code (HMAC) requests to Google Cloud Storage.

When you use the Google Cloud Storage XML API in a simple migration scenario, specifying the AWS signature identifier in the Authorization header lets Google Cloud Storage know to expect x-amz-* headers and Amazon S3 ACL XML syntax in your request.

Setting a default project

To use the Google Cloud Storage in a simple migration scenario, you must choose a default project. When you choose a default project, you are telling Google Cloud Storage the project to use for operations like GET service or PUT bucket.

To set a default project:

  1. Go to the Google Developers Console.

  2. Click the name of the project you would like to set as your default project.

  3. In the left sidebar, click Cloud Storage and then click Project Dashboard.

    This opens a new browser window that displays additional storage access configuration.

  4. In the left sidebar of the dashboard, ensure that the project selector displays the project you want as the default.

  5. In the left sidebar of the dashboard, ensure that the Storage Access pane is selected.

  6. Click the Make this my default project for interoperable storage access button.

    If the project is already the default project, you will see This is your default project for interoperable access.

This project is now your default project. You can change your default project at any time by choosing a different project and enabling interoperable access for it.

Back to top

Managing developer keys for a simple migration

To use the Google Cloud Storage XML API in a simple migration scenario, you must use keyed-hash message authentication code (HMAC) authentication with Google Cloud Storage developer keys. Developer keys consist of an access key and a secret. An access key is a 20 character alphanumeric string, which is linked to your Google account. All authenticated Google Cloud Storage requests, except those that use cookie-based authentication, must use an access key in the request so that the Google Cloud Storage system knows who is making the request. The following is an example of an access key:

GOOGTS7C7FUP3AIRVJTE

A secret is a 40 character Base-64 encoded string that is linked to a specific access key. A secret is a preshared key that only you and the Google Cloud Storage system know. You must use your secret to sign all requests as part of the authentication process. The following is an example of a secret:

bGoa+V7g/yqDXvKRqq+JTFn4uQZbPiQJo4pf9RzJ

To generate a developer key:

  1. Go to the Google Developers Console.

  2. Click the name of the project for which you would like to generate a developer key.

  3. In the left sidebar, click Cloud Storage and then click Project Dashboard.

    This opens a new browser window that displays additional storage access configuration.

  4. In the left sidebar of the dashboard, ensure that the project selector displays the project for which you want to generate a developer key.

  5. In the left sidebar of the dashboard, click Interoperable Access.

  6. Click Generate new key.

    The access key part of the new developer key is displayed.

  7. Click Show to show the secret part of the new developer key.

Security tips for working with developer keys

You can create up to five developer keys. This is useful if you are working on several different projects and you want to use different developer keys for each project.

Important: Never let another person use your developer keys. Your developer keys are linked to your Google account and you should treat them as you would any set of access credentials.

You can also use the key management tool to delete your developer keys and create new developer keys. You may want to do this if you think someone else is using your developer keys or you need to change your keys as part of a key rotation, which is a security best practice. If you have developer keys and you want to create new developer keys, we recommend that you first update your code with the new developer keys before you delete the old keys. When you delete developer keys, they become immediately invalid and they are not recoverable.

Back to top

Authenticating in a simple migration scenario

Authorization header

For operations in a simple migration scenario that require authentication, you will include an Authorization request header just like you do for requests to Amazon S3. The Authorization header syntax for an Amazon S3 request is:

Authorization: AWS AWS-ACCESS-KEY:signature

In a simple migration scenario, you only change the header to use your Google Developer access key:

Authorization: AWS GOOG-ACCESS-KEY:signature

The parts of the Authorization header are:

Signature identifier
The signature identifier identifies the signature algorithm and version that you are using. Using AWS indicates that you intend to send x-amz-* headers.
Access key
The access key identifies the entity that is making and signing the request. In a simple migration, replace the Amazon Web Service (AWS) access key ID you use to access Amazon S3 with your Google developer access key. Your Google developer access key starts with "GOOG".
Signature

The signature is a keyed cryptographic hash of various request headers. The signature is created by using HMAC-SHA1 as the hash function and your secret as the cryptographic key. The resulting digest is then Base64 encoded. When Google Cloud Storage receives your signed request, it uses the access key to look up your secret and verify that you created the signature. For more information on how to obtain an access and secret key, see Managing developer keys for access in a simple migration scenario.

In a simple migration, replace the AWS secret access key with your Google developer key secret as the cryptographic key.

Authentication calculation

This section describes the process of authenticating an XML API request in a simple migration scenario. While this section can be used to develop your own code to sign requests, it is mainly intended to be a review if you already have tools or libraries that sign your requests to Amazon S3. In this case, you will continue to use these tools to access Google Cloud Storage using the XML API, with the changes shown here.

Back to top

Access control in a simple migration scenario

To support simple migrations, Google Cloud Storage accepts ACLs produced by Amazon S3. In a simple migration scenario, you will use AWS as your signature identifier which tells Google Cloud Storage to expect ACL syntax using Amazon S3 ACL XML syntax. You should ensure that the Amazon S3 ACLs you use map to the Google Cloud Storage ACL model. For example, if your tools and libraries use Amazon S3's ACL syntax to grant bucket WRITE permission, then they must also grant bucket READ permission because Google Cloud Storage permissions are concentric. You do not need to specify both WRITE and READ permission when you grant WRITE permission using the Google Cloud Storage syntax.

Google Cloud Storage supports Amazon S3 ACL syntax in the following scenarios:

  • In a request to Google Cloud Storage to retrieve ACLs (for example, a GET Object or GET Bucket request), Google Cloud Storage returns Amazon S3 ACL syntax.
  • In a request to Google Cloud Storage to apply ACLs (for example, a PUT Object or PUT Bucket request), Google Cloud Storage expects to receive Amazon S3 ACL syntax.

The Authorization header in a simple migration scenario will use AWS for the signature identifier, but with your Google access key.

Authorization: AWS GOOG-ACCESS-KEY:signature

The following example shows a GET request to Google Cloud Storage to return the ACLs for an object.

GET europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Thu, 07 Nov 2013 01:04:05 GMT
Content-Type: application/xml
Authorization: AWS GOOG-ACCESS-KEY:Sku4/Nc1q4yJ6+7q7Sk3PFAAelE=

The response to the request includes the ACL using Amazon S3 ACL syntax.

<?xml version='1.0' encoding='UTF-8'?>
<AccessControlPolicy>
    <Owner>
        <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490
        </ID>
        <DisplayName>OwnerName</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
                xsi:type='CanonicalUser'>
                <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490</ID>
                <DisplayName>UserName</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>

The following example shows a PUT request to Google Cloud Storage to set the ACLs for an object. The example shows a request body with Amazon S3 ACL syntax.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Thu, 07 Nov 2013 01:55:16 GMT
Content-Type: application/xml
Content-Length: 337
Authorization: AWS GOOG-ACCESS-KEY:Ugm1lh6Bb+nAfNVa5ljUgfXYMfQ=

<?xml version='1.0' encoding='utf-8'?>
<AccessControlPolicy>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="AmazonCustomerByEmail">
        <EmailAddress>jane@gmail.com</EmailAddress>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Finally, in a simple migration scenario, you can also use the GOOG1 signature identifier in the Authorization header. In this case, you must use the Google Cloud Storage ACL syntax and ensure that all of your headers are Google headers, x-goog-*. While this is possible, we recommend that you move to a full migration as described below in order to realize all the benefits of Google Cloud Storage.

Back to top

Full migration

A full migration from Amazon S3 to Google Cloud Storage enables you to take advantage of all the features of Google Cloud Storage including:

Support for service accounts
Service accounts are useful for server-to-server interactions that do not require end-user involvement. For more information, see Service Accounts.
Support for multiple projects
Multiple projects allow you to have, in effect, many instances of the Google Cloud Storage service. This allows you to separate different functionality or services of your application or business as needed. For more information, see Using Projects.
OAuth 2.0 authentication
OAuth 2.0 relies on SSL for security instead of requiring your application to do cryptographic signing directly, and is easier to implement. With OAuth, your application can request access to data associated with a user's Google Account, and access can be scoped to several levels, including read-only, read-write, and full-control. For more information, see OAuth 2.0 Authentication.

To migrate fully from Amazon S3 to Google Cloud Storage, you will need to make the following changes:

  • Change any existing x-amz-* headers to corresponding x-goog-* headers.
  • Change AWS Access Control List (ACL) XML to the corresponding Google Cloud Storage ACL XML (see Specifying Bucket and Object ACLs).
  • Set the x-goog-project-id header in your requests. Note that in a simple migration scenario, you chose a default project for all requests. This is not needed in a full migration.
  • Get set up to use OAuth 2.0 authentication as described in OAuth 2.0 Authentication. The first step is to register your application (where you will be issuing requests from) with Google. Using OAuth 2.0 means that your Authorization header will look like this:
    Authorization: Bearer <oauth access token>

Back to top

Access control in a full migration

This section shows a few examples of access control to help you migrate from Amazon S3 to Google Cloud Storage. For an overview of access control in Google Cloud Storage, see Access Control.

In Google Cloud Storage, there are several ways to apply ACLs to buckets and objects (see Specifying Bucket and Object ACLs). Two of the ways you specify ACLs are analogous to what you do in Amazon S3:

  • The acl query string parameter to apply ACLs for specific scopes.
  • The x-goog-acl request header lets you apply predefined ACLs, which are sometimes known as canned ACLs.

Using the acl query string parameter

You can use the acl query string parameter for a Google Cloud Storage request exactly the same way you would use it for an Amazon S3 request. The acl parameter is used in conjunction with the PUT method to apply ACLs to the following: an existing object, an existing bucket, or a bucket you are creating. When you use the acl query string parameter in a PUT request, you must attach an XML document (using Google Cloud Storage ACL syntax) to the body of your request. The XML document contains the individual ACL entries that you want to apply to the bucket or object.

The following example shows a PUT request to Amazon S3 that uses the acl query string parameter. ACLs are defined in an XML document sent in the request body. The PUT request changes the ACLs on an object named europe/france/paris.jpg that is in a bucket named my-travel-maps. The ACL grants jane@gmail.com FULL_CONTROL permission.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 19:28:18 GMT
Content-Length: 598
Content-Type: application/xml
Authorization: AWS AWS-ACCESS-KEY:eNB6IUnqzsVAZ69TZZ5yFMQ5SvE=

<?xml version='1.0' encoding='utf-8'?>
<AccessControlPolicy>
  <Owner>
    <ID>5a6557ba40f7c86496ffceae789fcd888abc1b62a7149873a0fe12c0f60a7d95</ID>
    <DisplayName>ownerEmail@example.com</DisplayName>
  </Owner>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
        <ID>fd447671d60b979f78ee6fcec7b22afc80e6b26a4db16eed01afb8064047949b</ID>
        <DisplayName>jane@gmail.com</DisplayName>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>
Here is the same request to Google Cloud Storage:
PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 19:37:33 GMT
Content-Length: 268
Content-Type=application/xml
Authorization: Bearer <oauth access token>

<?xml version='1.0' encoding='utf-8'?>
<AccessControlList>
  <Entries>
  <Entry>
    <Permission>FULL_CONTROL</Permission>
    <Scope type="UserByEmail">
      <EmailAddress>jane@gmail.com</EmailAddress>
    </Scope>
  </Entry>
  </Entries>
</AccessControlList>

Note that Google Cloud Storage does not require an <Owner/> element in the ACL XML document. For more information, see Default Object ACLs.

You can also retrieve bucket and object ACLs by using the acl query string parameter with the GET method. The ACLs are described in an XML document, which is attached to the body of the response. You must have FULL_CONTROL permission to apply or retrieve ACLs on an object or bucket.

Applying ACLs with an extension request header

You can use the x-goog-acl header in a Google Cloud Storage request to apply predefined ACLs to buckets and objects exactly the same way you would use the x-amz-acl header in an Amazon S3 request. You typically use the x-goog-acl (x-amz-acl) header to apply a predefined ACL to a bucket or object when you are creating or uploading the bucket or object. The Google Cloud Storage predefined ACLs are similar to Amazon S3 Canned ACLs, including private, public-read, public-read-write, as well as others. For a list of Google Cloud Storage predefined ACLs, see Predefined ACLs.

The following example shows a PUT Object request that applies the public-read ACL to an object named europe/france/paris.jpg that is being uploaded into a bucket named my-travel-maps in Amazon S3.

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 20:48:42 GMT
Content-Length: 888814
Content-Type: image/jpg
x-amz-acl: public-read
Authorization: AWS AWS-ACCESS-KEY:o8PRLyrTxsMEznHzgQvTt2TnWTI=

<888814 bytes in entity body>

Here is the same request to Google Cloud Storage:

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 20:49:57 GMT
Content-Length: 888814
Content-Type: image/jpg
x-goog-acl: public-read
Authorization: Bearer <oauth access token>

<888814 bytes in entity body>

You can also use the x-goog-acl header to apply a predefined ACL to an existing bucket or object. To do this, include the acl query string parameter in your request but do not include an XML document in your request. Applying a predefined ACL to an existing object or bucket is useful if you want to change from one predefined ACL to another, or you want to update custom ACLs to a predefined ACL. For example, the following PUT Object request applies the predefined ACL private to an object named europe/france/paris.jpg that is in a bucket named my-travel-maps.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 00:26:36 GMT
Content-Length: 0
x-goog-acl: private
Authorization: Bearer <oauth access token>

<empty entity body>

For more information about managing ACLs, see Managing Access Control.

Migrating from Amazon S3 to Google Cloud Storage Request Methods

Google Cloud Storage supports the same standard HTTP request methods for reading and writing data to your buckets as are supported in Amazon S3. Therefore, the majority of your tools and libraries that you currently use with Amazon S3, will work as is with Google Cloud Storage. Google Cloud Storage supports the following request methods:

  • Service request for GET.
  • Bucket requests, including PUT, GET, DELETE.
  • Object requests, including GET, POST, PUT, HEAD, and DELETE.

For more information, see XML API Reference Methods. Keep in mind that when you send requests to Google Cloud Storage, you will need to change the request body, when applicable, to use the appropriate Google Cloud Storage syntax. For example, when you create a lifecycle configuration for a bucket, use the Google Cloud Storage lifecycle XML, which is different than the Amazon S3 lifecycle XML.

There are a few differences between Google Cloud Storage XML API and Amazon S3 which are summarized below, with suggested Google Cloud Storage alternatives:

Amazon S3 Functionality Google Cloud Storage XML API Functionality
Multipart upload.
POST /<object-name>, PUT /<object-name>

A multipart upload in the Google Cloud Storage XML API is called a resumable upload.

Alternatively, you can use the multipart upload feature of the JSON API.

GET/POST bucket query string parameters:
  • "policy" - Working with Amazon S3 bucket policies.
  • "website" - Configuring bucket websites.
  • "tagging" - Tagging buckets for cost allocation purposes.
  • "notification" - Notifying for bucket events.
  • "requestPayment" - Configuring who pays for the request and the data download from a bucket.
Alternatives:
  • "policy" - Google Cloud Storage ACLs, project team membership, and the ability to use multiple projects address many of the scenarios where bucket policies are used.
  • "website" - Use the gsutil web command to manage websites, or try the JSON API (see buckets resource).
  • "tagging" - Use multiple projects to track different cost centers. For more about projects, see Managing Projects.
  • "notification" - Use JSON API object notification.
  • "requestPayment" - Use multiple projects with different billing profiles to manage who pays for requests and downloaded data from a bucket. For more about configuring billing, see Billing in the Google APIs Console Help documentation.
Multiple object delete.
POST /?delete

Use the gsutil rm command to easily remove multiple objects. The rm command supports the "-m" option to perform parallel (multi-threaded/multi-processing) deletes.

Alternatively, the JSON API supports sending batch requests to reduce the number of HTTP connections your client makes.

Migrating from Amazon S3 to Google Cloud Storage Headers

Google Cloud Storage uses several standard HTTP headers as well as several custom (extension) HTTP headers. If you are transitioning from Amazon S3 to Google Cloud Storage, you can convert your custom Amazon S3 headers to the equivalent Google Cloud Storage custom header or similar functionality as shown in the table below.

Amazon S3 Header Google Cloud Storage Header
x-amz-acl x-goog-acl
x-amz-date x-goog-date
x-amz-meta-* x-goog-meta-*
x-amz-grant-* Use x-goog-acl with a predefined ACL value.
x-amz-copy-source x-goog-copy-source
x-amz-metadata-directive x-goog-metadata-directive
x-amz-copy-source-if-match x-goog-copy-source-if-match
x-amz-copy-source-if-none-match x-goog-copy-source-if-none-match
x-amz-copy-source-if-unmodified-since x-goog-copy-source-if-unmodified-since
x-amz-copy-source-if-modified-since x-goog-copy-source-if-modified-since
x-amz-server-side-encryption Not required. Google Cloud Storage automatically encrypts all data before it is written to disk. For more information, see Server-Side Encryption.
x-amz-storage-class You can specify storage class when you create a bucket. For more information, see Creating a Durable Reduced Availability Bucket.
x-amz-mfa Use OAuth 2.0 Authentication.
x-amz-website-redirect-location, x-amz-copy-source-range,
n/a

Back to top

Discussion groups and support for XML API compatibility with Amazon S3

The Google Cloud Storage gs-discussion group which formerly supported XML API interoperability and migration issues, is in archive mode. The discussion forum can now be accessed on Stack Overflow using the tag google-cloud-storage. See the Community section on the Resources and Support page for more information about discussion forums and subscribing to announcements.

Back to top

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.