Google Cloud Storage

Object Versioning


Google Cloud Storage buckets that have Object Versioning turned on maintain an archive of objects, providing a way to un-delete data that you accidentally deleted, or to retrieve older versions of your data. This page describes how Object Versioning works and provides examples for working with it.

Contents

How Object Versioning Works

Google Cloud Storage allows you to enable Object Versioning at the bucket level. Once enabled, a history of modifications (overwrite / delete) of objects is kept for all objects in the bucket. You can list archived versions of an object, restore an object to an older state, or permanently delete a version, as needed. All objects have generation properties associated with them to help you identify the objects and perform safe read-modify-write updates on them as well as perform conditional operations on them. Generation properties are numeric and monotonically increasing, which lets you quickly evaluate the version order. When an object is overwritten or deleted in a bucket which has versioning enabled, a copy of the object is automatically saved with generation properties that identify it. You can turn versioning on or off for a bucket at any time. Turning versioning off leaves existing object versions in place, and simply causes the bucket to stop accumulating new object versions. In this case, if you upload to an existing object, the current version is overwritten instead of creating a new version.

Object Versioning Details

Google Cloud Storage uses two generation properties, that together identify the version of an object. These properties are always present with every version of the object, even if versioning is not enabled. They can be used for conditional updates to enforce ordering of updates. See the Examples section below for details.

Google Cloud Storage marks every object using the following properties:

generation
Identifies the content (data) generation, and is updated when the content of an object is overwritten.
metageneration
Identifies the metadata generation, and is updated every time the metadata (e.g. ACL) for a given content generation is updated. metageneration also gets reset to 1 whenever generation changes. The metageneration has no meaning without the content generation and should only be used in conjunction with it. In other words, it is meaningless to compare metadata generations of two versions that have different data generations.

For more details about working with Object Versioning, see the Tips section below.

Object Versioning Examples

Let's take a look at what happens when you create objects, overwrite object data, or update objects on a versioning enabled bucket. In the following examples, we show you how to work with versioning using the gsutil tool, the Java client library, and the XML API. You can read and write data to a bucket that has versioning enabled with any tool that supports Google Cloud Storage.

Enabling Versioning

The following code sample shows you how to enable versioning on a bucket.

gsutil

Uses the gsutil tool.

gsutil versioning set on gs://bucket

Enabling versioning for gs://bucket/...

To disable versioning, substitute off for on in the command above.

Java

Uses the Java client library.

Bucket content = new Bucket().setVersioning(new Versioning().setEnabled(true));
Bucket versionedBucket = storage.buckets().patch("bucket", content).execute();
System.out.println("versioning enabled: " + versionedBucket.getVersioning().get("enabled"));
versioning enabled: true

To disable versioning, substitute false for true in the setEnabled method.

XML API

Uses the XML API.

PUT /bucket/?versioning HTTP/1.1
Host: storage.googleapis.com
Accept-Encoding: identity
Date: Fri, 15 Feb 2013 00:06:00 GMT
Content-Length: 114

<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>
HTTP/1.1 200 OK

To disable versioning, use a empty VersioningConfiguration element, <VersioningConfiguration/>.

The following code sample shows you how to verify the current status of versioning on a bucket.

gsutil

Uses the gsutil tool.

gsutil versioning get gs://bucket

gs://bucket: Enabled

Java

Uses the Java client library.

Bucket bucket = storage.buckets().get("bucket").execute();
System.out.println("versioning enabled: " + bucket.getVersioning().get("enabled"));
versioning enabled: true

XML API

Uses the XML API.

GET /bucket/?versioning HTTP/1.1
Host: storage.googleapis.com
Date: Fri, 15 Feb 2013 00:07:43 GMT
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 6 2013 15:53:54 (1360194834)
Content-Length: 113
Date: Fri, 15 Feb 2013 00:07:43 GMT
Expires: Fri, 15 Feb 2013 00:07:43 GMT

<?xml version='1.0' encoding='UTF-8'?><VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>

Creating a New Object

When you create a new object, generation and metageneration properties are created for the object regardless of whether versioning on a bucket is enabled or not.

The following code sample shows creating an object for which the generation for the object is 1360887697105000 and metageneration is 1.

gsutil

Uses the gsutil tool.

echo 'this is foo' > foo
gsutil cp -v foo gs://bucket

Copying file://foo [Content-Type=application/octet-stream]...
Created: gs://bucket/foo#1360887697105000

Java

Uses the Java client library.

InputStream inputStream = new ByteArrayInputStream("this is foo".getBytes());
InputStreamContent mediaContent = new InputStreamContent("text/plain", inputStream);
StorageObject fooObject = storage.objects().insert("bucket", null, mediaContent).setName("foo").execute();
System.out.println("generation: " + fooObject.getGeneration());
generation: 1360887697105000

XML API

Uses the XML API.

PUT /bucket/foo HTTP/1.1
Host: storage.googleapis.com
Content-Length: 12
x-goog-acl: public-read
x-goog-api-version: 2

this is foo
HTTP/1.1 200 OK
ETag: "4be8bde80854190cbe801133c6682ecf"
x-goog-generation: 1360887697105000
x-goog-metageneration: 1

Overwriting Object Data

If you overwrite an object in a versioned bucket, that is, you upload a new version of the object, Google Cloud Storage archives the current object, creates the new object and increases the generation, as shown in the following figure.

Object Version Control for Data Update
Object Data Update

When you overwrite the data of an object by uploading over the existing object, the x-goog-generation is updated and x-goog-metageneration is restored back to 1. The following example shows how to overwrite an existing object.

gsutil

Uses the gsutil tool.

echo 'this is foo but better' > foo
gsutil cp -v foo gs://bucket

Copying file://foo [Content-Type=application/octet-stream]...
Created: gs://bucket/foo#1360887759327000

Java

Uses the Java client library.

InputStream inputStream = new ByteArrayInputStream("this is foo is better".getBytes());
InputStreamContent mediaContent = new InputStreamContent("text/plain", inputStream);
StorageObject fooObj = storage.objects().insert("bucket", null, mediaContent).setName("foo").execute();
System.out.println("generation: " + fooObj.getGeneration());
generation: 1360887759327000

XML API

Uses the XML API.

PUT /bucket/foo HTTP/1.1
Host: storage.googleapis.com
Content-Length: 23
x-goog-acl: public-read
x-goog-api-version: 2

this is foo but better
HTTP/1.1 200 OK
ETag: "b859c7443ae764f0cbeba4647e3831d4"
x-goog-generation: 1360887759327000
x-goog-metageneration: 1

If you set the x-goog-if-generation-match header to 0 when uploading an object, Google Cloud Storage only performs the specified request if the object does not currently exist. For example, you can perform a PUT request to create a new object with x-goog-if-generation-match set to 0, and the object will be created only if it doesn't already exist. Otherwise, Google Cloud Storage aborts the update with a status code of 412 Precondition Failed.

The following examples shows how to use the x-goog-if-generation-match header. Note that even though the content of the file to upload is different, the update does not occur.

gsutil

Uses the gsutil tool.

echo 'another bar' > foo
gsutil -h 'x-goog-if-generation-match:0' cp -v foo gs://bucket

Copying file://foo [Content-Type=application/octet-stream]...
GSResponseError: status=412, code=PreconditionFailed, reason=Precondition Failed.

Java

Uses the Java client library.

InputStream inputStream = new ByteArrayInputStream("another bar".getBytes());
InputStreamContent mediaContent = new InputStreamContent("text/plain", inputStream);
StorageObject fooObj = storage.objects().insert("bucket", null, mediaContent)
    .setIfGenerationMatch(new BigInteger("0"))
    .setName("foo").execute();
412 Precondition Failed

XML API

Uses the XML API.

PUT /bucket/foo HTTP/1.1
x-goog-if-generation-match:0
Content-Length: 11

another bar
HTTP/1.1 412 Precondition Failed
<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>PreconditionFailed</Code>
    <Message>At least one of the pre-conditions you specified did not hold.</Message>
</Error>

However, the request will succeed if archived generations of the object exist but there is no current live object. In other words, setting the x-goog-if-generation-match header to 0 will only succeed if attempting a non-versioned read of the object would result in a 404 Not Found (not found) error.

Listing Archived Object Versions

At this point, there are now two generations of the same object: the latest generation of the object, which is the current one with generation 1360887759327000, and one archived object with generation 1360887697105000. If you do not specify that object version should be returned when listing objects, you only get the current object as shown in the following example.

gsutil

Uses the gsutil tool.

gsutil ls -l gs://bucket

23  2013-02-15T00:22:39.324Z  gs://bucket/foo

Java

Uses the Java client library.

Storage.Objects.List listObjects = storage.objects().list(settings.getBucket());
Objects objects;
do {
  objects = listObjects.execute();
  for (StorageObject obj : objects.getItems()) {
    System.out.format("Size: %s, Creation Date: %s, Name: %s\n",
        obj.getSize(), obj.getUpdated(), obj.getName());
  }
  listObjects.setPageToken(objects.getNextPageToken());
} while (null != objects.getNextPageToken());
Size: 23, Creation Date: 2013-02-15T00:22:39.324Z, Name: foo

XML API

Uses the XML API. The output is formatted for reading clarity.

GET /bucket HTTP/1.1
HTTP/1.1 200 OK

<?xml version='1.0' encoding='UTF-8'?>
  <ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
    <Name>bucket</Name>
    <Prefix></Prefix>
    <Marker></Marker>
    <IsTruncated>false</IsTruncated>
    <Contents>
      <Key>foo</Key>
      <Generation>1360887759327000</Generation>
      <MetaGeneration>1</MetaGeneration>
      <LastModified>2013-02-15T00:22:39.324Z</LastModified>
      <ETag>"b859c7443ae764f0cbeba4647e3831d4"</ETag>
      <Size>23</Size>
      <Owner>
        <ID>00b4903a97a3341225a8da631155093237119370d36b57e9d73bd3475050babb</ID>
        <DisplayName>Example User</DisplayName>
      </Owner>
    </Contents>
  </ListBucketResult>

However, if you specify that versions should be returned when listing objects, then all versions of objects are returned as shown in the following example. Note that how you indicate this depends on how you access Google Cloud Storage. In the XML API example below, you specify the versions query parameter. In the gsutil example, you specify the -a option.

gsutil

Uses the gsutil tool.

gsutil ls -la gs://bucket

 6  2013-02-15T00:21:37.102Z  gs://bucket/foo#1360887697105000  metageneration=1
23  2013-02-15T00:22:39.324Z  gs://bucket/foo#1360887759327000  metageneration=1

Java

Uses the Java client library.

Storage.Objects.List listObjects = storage.objects().list(settings.getBucket()).setVersions(true);
Objects objects;
do {
  objects = listObjects.execute();
  for (StorageObject obj : objects.getItems()) {
    System.out.format("Size: %s, Creation Date: %s, Name: %s\n",
        obj.getSize(), obj.getUpdated(), obj.getName());
  }
  listObjects.setPageToken(objects.getNextPageToken());
} while (null != objects.getNextPageToken());
Size: 6, Creation Date: 2013-02-15T00:21:37.102Z, Name: foo
Size: 23, Creation Date: 2013-02-15T00:22:39.324Z, Name: foo

XML API

Uses the XML API.

GET /bucket?versions HTTP/1.1
HTTP/1.1 200 OK

<?xml version='1.0' encoding='UTF-8'?>
  <ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
    <Name>bucket</Name>
    <Prefix></Prefix>
    <Marker></Marker>
    <GenerationMarker></GenerationMarker>
    <IsTruncated>false</IsTruncated>
    <Version>
      <Key>foo</Key>
      <Generation>1360887697105000</Generation>
      <MetaGeneration>1</MetaGeneration>
      <IsLatest>false</IsLatest>
      <LastModified>2013-02-15T00:21:37.102Z</LastModified>
      <DeletedTime>2013-02-15T00:22:39.327Z</DeletedTime>
      <ETag>"4be8bde80854190cbe801133c6682ecf"</ETag>
      <Size>12</Size>
      <Owner>
        <ID>00b4903a97a3341225a8da631155093237119370d36b57e9d73bd3475050babb</ID>
        <DisplayName>Example User</DisplayName>
      </Owner>
    </Version>
    <Version>
      <Key>foo</Key>
      <Generation>1360887759327000</Generation>
      <MetaGeneration>1</MetaGeneration>
      <IsLatest>true</IsLatest>
      <LastModified>2013-02-15T00:22:39.324Z</LastModified>
      <DeletedTime></DeletedTime>
      <ETag>"b859c7443ae764f0cbeba4647e3831d4"</ETag>
      <Size>23</Size>
      <Owner>
        <ID>00b4903a97a3341225a8da631155093237119370d36b57e9d73bd3475050babb</ID>
        <DisplayName>Example User</DisplayName>
    </Owner>
    </Version>
  </ListBucketResult>
There are a few differences in the results of the GET request when using the versions query parameter compared to not using it. Specifically, Google Cloud Storage returns the following when you provide a version query parameter in your request:
  • A Version element which contains information about each object.
  • A DeletedTime element which contains the time the object was archived (deleted or overwritten).
  • An IsLatest element which indicates if the specific object is the latest version.
  • Although it is not shown in the listing above, a NextGenerationMarker element is returned if the listing of objects is a partial listing. You could then use this marker with the NextMarker value to list more results. You will get a partial list of results when you have many object versions in a bucket. In this case, the results are truncated and you use the NextGenerationMarker value to pick up where you left off on subsequent requests with a generationmarker query parameter. Use the generationmarker query parameter like you use the marker query parameter to page through a listing for a nonversioned bucket.

Continuing with the example, you can verify that you can get both generations of the object by first specifying no generation, which fetches the current version of the object, and then explicitly specifying the generation of the current object and then the archived generation as shown in the following examples.

gsutil

Uses the gsutil tool.

gsutil cat gs://bucket/foo

this is foo but better
gsutil cat gs://bucket/foo#1360887759327000

this is foo but better
gsutil cat gs://bucket/foo#1360887697105000

this is foo

Java

Uses the Java client library.

Storage.Objects.Get fooObj = storage.objects().get("bucket", "foo");
ByteArrayOutputStream out = new ByteArrayOutputStream();
fooObj.executeMediaAndDownloadTo(out);
System.out.println(out);
this is foo but better
Storage.Objects.Get fooObj = storage.objects().get("bucket", "foo");
fooObj.setGeneration(new BigInteger("1360887759327000"));
ByteArrayOutputStream out = new ByteArrayOutputStream();
fooObj.executeMediaAndDownloadTo(out);
System.out.println(out);
this is foo but better
Storage.Objects.Get fooObj = storage.objects().get("bucket", "foo");
fooObj.setGeneration(new BigInteger("1360887697105000"));
ByteArrayOutputStream out = new ByteArrayOutputStream();
fooObj.executeMediaAndDownloadTo(out);
System.out.println(out);
this is foo

XML API

Uses the XML API.

GET /bucket/foo HTTP/1.1
HTTP/1.1 200 OK
ETag: "b859c7443ae764f0cbeba4647e3831d4"
x-goog-generation: 1360887759327000
x-goog-metageneration: 1

this is foo but better
GET /bucket/foo?generation=1360887759327000 HTTP/1.1
HTTP/1.1 200 OK
ETag: "b859c7443ae764f0cbeba4647e3831d4"
x-goog-generation: 1360887759327000
x-goog-metageneration: 1

this is foo but better
GET /bucket/foo?generation=1360887697105000 HTTP/1.1
HTTP/1.1 200 OK
x-goog-generation: 1360887697105000
x-goog-metageneration: 1

this is foo

Updating Object Metadata

If you update object metadata, for example, update an object's ACL, Google Cloud Storage does not archive a copy of the current object, but simply updates the metadata of the specified generation (or live object if none given) and increases the metadata generation by 1. The generation of the object remains unchanged. To see that, let's update the ACLs on the archived object:

gsutil

Uses the gsutil tool.

gsutil -h 'x-goog-if-generation-match:1360887697105000' \
  acl set public-read gs://bucket/foo

Setting ACL on gs://bucket/foo...
gsutil -d cat gs://bucket/foo#1360887697105000

reply: 'HTTP/1.1 200 OK\r\n'
header: x-goog-generation: 1360887697105000
header: x-goog-metageneration: 2

this is foo

Java

Uses the Java client library.

StorageObject content = new StorageObject()
    .setName("foo")
    .setAcl(ImmutableList.of(
        new ObjectAccessControl().setEntity("allUsers").setRole("READER")
        ));
Storage.Objects.Patch fooObj = storage.objects().patch("bucket", "foo", content);
fooObj.setIfGenerationMatch(new BigInteger("1360887697105000"));
StorageObject retObj = fooObj.execute();
System.out.println("generation: " + retObj.generation());
System.out.println("metageneration: " + retObj.getMetageneration());
generation: 1360887697105000
metageneration 2
Storage.Objects.Get fooObj = storage.objects().get("bucket", "foo");
fooObj.setGeneration(new BigInteger("1360887697105000"));
ByteArrayOutputStream out = new ByteArrayOutputStream();
fooObj.executeMediaAndDownloadTo(out);
System.out.println(out);
this is foo

XML API

Uses the XML API.

PUT /bucket/foo?acl&generation=1360887697105000 HTTP/1.1
x-goog-acl: public-read
HTTP/1.1 200 OK
x-goog-generation: 1360887697105000
x-goog-metageneration: 2
GET /bucket/foo?generation=1360887697105000 HTTP/1.1
HTTP/1.1 200 OK
ETag: "4be8bde80854190cbe801133c6682ecf"
x-goog-generation: 1360887697105000
x-goog-metageneration: 2

this is foo

And you can see from the example, the metageneration has been changed but the generation has not, since the data has not changed.

Conditional Updates Using Object Versioning

To ensure you are updating the correct ACL (when doing Read-Modify-Write), you should use both x-goog-if-generation-match and x-goog-if-metageneration-match. as shown in the following example. The example shows how to change the ACL on an object if the generation and metadata generation match specified values and then confirm that only the metadata generation changed by getting the object.

gsutil

Uses the gsutil tool.

gsutil -h 'x-goog-if-generation-match:1360887697105000' \
  -h 'x-goog-if-metageneration-match:2' acl set private gs://bucket/foo

Setting ACL on gs://bucket/foo...
gsutil -d cat gs://bucket/foo#1360887697105000

reply: 'HTTP/1.1 200 OK\r\n'
header: x-goog-generation: 1360887697105000
header: x-goog-metageneration: 3

this is foo

Java

Uses the Java client library.

StorageObject content = new StorageObject()
    .setName("foo")
    .setAcl(ImmutableList.of(
        new ObjectAccessControl().setEntity("allUsers").setRole("READER")
        ));
Storage.Objects.Patch fooObj = storage.objects().patch("bucket", "foo", content);
fooObj.setIfGenerationMatch(new BigInteger("1360887697105000"));
fooObj.setIfMetagenerationMatch(new BigInteger("2"));
StorageObject retObj = fooObj.execute();
System.out.println("generation: " + retObj.generation());
System.out.println("metageneration: " + retObj.getMetageneration());
generation: 1360887697105000
metageneration 2
Storage.Objects.Get fooObj = storage.objects().get("bucket", "foo");
fooObj.setGeneration(new BigInteger("1360887697105000"));
ByteArrayOutputStream out = new ByteArrayOutputStream();
fooObj.executeMediaAndDownloadTo(out);
System.out.println(out);
this is foo

XML API

Uses the XML API.

PUT /bucket/foo?acl HTTP/1.1
x-goog-acl: private
x-goog-if-generation=1360887697105000
x-goog-if-metageneration: 2
HTTP/1.1 200 OK
x-goog-generation: 1360887697105000
x-goog-metageneration: 3
GET /bucket/foo?generation=1360887697105000 HTTP/1.1
HTTP/1.1 200 OK
ETag: "4be8bde80854190cbe801133c6682ecf"
x-goog-generation: 1360887697105000
x-goog-metageneration: 3

this is foo

Copying Objects

You can copy versioned objects just like unversioned ones, you just have to specify which generation you're from which to copy. For example, when you use gsutil you specify the generation appended to the URI as shown below. When you use the XML API you specify the generation with a x-goog-copy-source-generation header.

gsutil

Uses the gsutil tool.

gsutil cp gs://bucket/foo#1360887697105000 gs://bucket/newfoo

Copying gs://bucket/foo#1360887697105000...

Java

Uses the Java client library.

Storage.Objects.Copy newfooObj = storage.objects()
    .copy("bucket", "foo", "bucket", "newfoo", null)
    .setSourceGeneration(new BigInteger("1360887697105000"));
newfooObj.execute();

XML API

Uses the XML API.

PUT /bucket/newfoo HTTP/1.1
x-goog-copy-source:/bucket/foo
x-goog-copy-source-generation:1360887697105000
Content-Length: 0
HTTP/1.1 200 OK

<?xml version='1.0' encoding='UTF-8'?>
<CopyObjectResult>
    <LastModified>2013-02-15T00:21:37.102Z</LastModified>
    <ETag>"4be8bde80854190cbe801133c6682ecf"</ETag>
</CopyObjectResult>

Conditional Copies Using Object Versioning

To ensure you are copying from the correct object, you should use x-goog-copy-source-generation and/or and x-goog-copy-source-metageneration. To ensure that you're copying into (or updating) the correct objects metadata, you should use either x-goog-if-generation-match and/or and x-goog-if-metageneration-match.

Tips

This section discusses tips to help you work with Object Versioning more effectively.

Using gsutil

  • The gsutil tool has comprehensive support for working with versioned objects that makes many tasks involving object versioning easier. For example, you can work with versioned objects directly by appending the generation you want to work with to the object name with a "#". For more information, see Object Versioning and Concurrency Control or view the built-in documentation by running:
    gsutil help versions
    

Creating, retrieving, and copying versioned objects

  • Consider using Object Versioning for conditional updates instead of ETags because versioning provides a stronger guarantee than ETags by keeping track of all updates to the object, including ACL changes.
  • When you create an object, Google Cloud Storage sets the generation property according to a server-side algorithm such that it is higher (monotonically increasing) than all existing values for this property on archived versions of the object. The newly created object's metageneration property is set to 1.
  • You can work with a specific archived object by specifying its generation in your request.
  • If you do not specify a generation in your request, Google Cloud Storage returns the current live object, or 404 Not Found if there is no current live object (even if there are archived objects).
  • You can copy an archived object version to the current live version. To do this, send a copy request with the archived generation specified by the x-goog-copy-source-generation request header. Note that when you do this, it creates a new object version, which will incur additional storage charges. To prevent the additional charge, you can delete the archived copy if it is no longer needed.
  • Archived objects retain the metadata associated with their latest metageneration. Most importantly, an archived version retains its ACLs and does not necessarily exhibit the same permissions as the live version of the object.

Using preconditions

  • If you request a generation of an object that does not exist, Google Cloud Storage returns a 404 Not Found. If you specify the x-goog-if-generation-match request header, without the object generation, then Google Cloud Storage returns the current live object only if the generation of the object exists; otherwise, Google Cloud Storage aborts the operation and returns 412 Precondition Failed.

  • You can update metadata for an archived object by specifying its generation in your request. To ensure safe read-modify-write semantics, you should use the x-goog-if-metageneration-match request header. Using this header will cause the update to fail if the metadata you are attempting to update was changed between the time you read the metadata and sent the update).
  • If you make several concurrent requests with the x-goog-if-generation-match header, Google Cloud Storage's strong consistency will only allow one of those several requests to succeed, which can be useful if your objects are updated from several sources and you need to ensure that users don't accidentally override them.

Deleting versioned objects

  • You can delete an archived object by specifying its generationin a delete request. Google Cloud Storage deletes the archived object associated with that generation permanently.
  • If you send a delete request without specifying a generation, Google Cloud Storage archives the current live object and causes it to appear missing to subsequent version-unaware requests. You can retrieve the archived object by specifying its generation in your retrieval request. You can delete the version only by specifying its generation in a delete request.
  • If you send a delete request with a generation that corresponds to the currently live object, Google Cloud Storage deletes the object without making an archived copy.

Back to top

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.