Google Cloud Storage

Hello Google Cloud Storage

The Hello Google Cloud Storage! page provides a quick and easy way to get started using Google Cloud Storage through the gsutil tool. These exercises walk you through some basic tasks that can be done with Google Cloud Storage using gsutil and help you explore basic Google Cloud Storage functionality.

At the end of these exercises, you can read the gsutil built-in help for more advanced gsutil functionality, by running the command:

gsutil help

Note: The exercises in this document require that you set up gsutil and authenticate to the Google Cloud Storage service. If you are simply downloading or accessing publicly-accessible objects, you can go straight to installing the gsutil client and downloading your objects. You do not need to activate Google Cloud Storage, turn on billing, or authenticate to the Google Cloud Storage service to access publicly-accessible objects.

Prerequisites

Before you can start the exercises though, you need to fulfill the following prerequisites:

  • You must have access to Google Cloud Storage to complete these exercises.

    You can access Google Cloud Storage if you were added as a project team member to an existing Google Account or if you have recently created a Google Cloud Storage account.

  • You must have Python 2.6 or Python 2.7 installed.

    The current version of gsutil only supports Python 2.6 or Python 2.7.

  • You must have the newest version of gsutil installed.

    gsutil is a Python application that lets you access Google Cloud Storage.

After you have fulfilled the prerequisites above, continue to the exercises.

Hello Google Cloud Storage! Exercises

The following exercises help you get started using Google Cloud Storage through the gsutil command line tool. If you have not reviewed the prerequisites for starting this exercise, please do so above.

By the end of this exercise, you will have completed the following tasks:

  • Create and delete a bucket
  • Upload, delete and move objects
  • List your buckets and objects
  • Share objects and buckets

Step 1: Create a bucket

First, create two buckets in your project with unique bucket names. You can do this by using the mb command. For example, the following command creates buckets named cats and dogs:

gsutil mb gs://cats gs://dogs

Note: If you are running gsutil on a Windows operating system you must first call the Python interpreter. For example, to create two buckets named cats and dogs you must type python gsutil mb gs://cats gs://dogs at the command prompt.

Choose bucket names that are unique when you create these buckets because it is likely that if you try to create buckets named cats or dogs, you will receive the following error:

Failure: GSCreateError: 409 Conflict
BucketNameUnavailableThe requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again..

It means that the bucket names cat or dog are already taken. Google Cloud Storage has a single namespace so your bucket names must be unique across the entire Google Cloud Storage system. Someone else has already created buckets named cats or dogs, so you won't be allowed to create buckets with those names. You'll have to modify the bucket names so that they are unique, but you must also follow the bucket naming guidelines.

It is also possible to specify a location constraint so that your bucket is created in a geographical location by including the -l <location> flag. For more information, see Specifying Bucket Locations.

Step 2: Upload objects into a bucket

Now that you have some buckets, try uploading a couple of objects into one of them. To do this, create a new directory, add two image files to the directory, and name the files poodle.jpg and collie.jpg. To upload the files, change to the directory you just created and use the cp command as follows:

gsutil cp *.jpg gs://dogs

The cp command behaves much like the Unix cp command with the recursion (-R) option, allowing you to copy whole directories or just the contents of directories. gsutil also supports wildcards, which makes it easy for you to copy or move batches of files. Depending on how you upload your objects, gsutil creates new objects with names that follow certain guidelines:

  • If you use the -R option to recursively upload a directory, gsutil creates your object names based on the directory where it starts recursive processing.

    For example, if you have a directory located at client1/low_priority/ and you recursively copy the directory using gsutil cp -R client1/low_priority dst_uri, gsutil creates new objects with low_priority in the object name, because low_priority is the directory where the recursive processing begins:

    gs://dst_uri/low_priority/my_object

    The -R option also copies subdirectories and includes subdirectory names in the object name. For example, if client1/low_priority contained another directory named datasheets, gsutil cp -R client1/low_priority dst_uri will create objects named:

    gs://dst_uri/low_priority/building.cad
    gs://dst_uri/low_priority/schematics.cad
    gs://dst_uri/low_priority/datasheets/assets.txt
    gs://dst_uri/low_priority/datasheets/expenses.txt
    

    This can be useful for organizational purposes and for use with the Google Developers Console, but if you aren't careful, you could also embed unintended paths in file names (e.g., embedding low_priority for a customer to see in the example above).

  • If you need to copy or upload a directory without prepending the directory name, you can use wildcards.

    For example, if you copy files using gsutil cp client1/low_priority/* dst_uri, gsutil copies the objects in the directory without prepending the directory name:

    gs://dst_uri/my_object

Step 3: List the buckets and objects

Next, list the buckets and objects that you created. You can use the ls command to do this. For example, the following command lists your buckets in a specified project:

gsutil ls

The following command lists the objects that are stored in the bucket dogs:

gsutil ls gs://dogs

Using the -l option

You can use the -l option to print a long listing of your objects and buckets. This provides more information than the ls command alone. For example, the following command lists information about all the objects inside dogs:

gsutil ls -l gs://dogs

If you want to get information about a particular bucket instead of its contents, you can include the -b option followed by the bucket name. For example, the following command lists information about the bucket dogs:

gsutil ls -l -b gs://dogs

To list the total size of all your buckets, you can use either of the following commands:

  • gsutil ls -l

    Provides a listing of your buckets and their total size

  • gsutil ls -l gs://*

    Provides a complete list of all your objects in every bucket and the total size

Using the -L option

You can use the -L option with the ls command to list the most detailed information available about your objects and buckets in your project. For example, the following command provides comprehensive information such as the object size, last modified date, and cache control for all objects inside the dogs bucket:

gsutil ls -L gs://dogs/*

You can view detailed information about your buckets by using the following command:

gsutil ls -L

If you want detailed information about a particular bucket instead of its contents, you can include the -b option followed by the bucket name. For example, the following command provides detailed information about the bucket dogs:

gsutil ls -L -b gs://dogs

Note: Although this option gives the most detailed information about your objects and buckets, it can also use more of your monthly bandwidth quota because the -L option uses multiple GET requests to retrieve all the information it needs. In comparison, the -l option will only perform one GET request for the information it needs, and is much faster than the -L option.

Step 4: Move an object

You can move objects from one bucket to another by using the mv command. The mv command can also be used to rename an object. To move all of the files that are in the dogs bucket into the cats bucket, use the following command:

gsutil mv gs://dogs/*.jpg gs://cats/

To change the name of poodle.jpg, which you just moved into the cats bucket, you use the following command:

gsutil mv gs://cats/poodle.jpg gs://cats/siamese.jpg

Step 5: Download an object

You can also download objects by using the cp command. To download the objects that are in the cats bucket and save them in a directory named pets that's in your local directory, use the following command:

gsutil cp gs://cats/*.jpg file://pets/

Step 6: Sharing objects and buckets

ACLs define who has which kinds of permissions to access an object or bucket. Google Cloud Storage lets you grant permissions for a wide range of entities, such as individual users, Google Apps domains, and Google groups. You can share objects and buckets with other people by using the acl ch command to modify the ACLs that are applied to objects or buckets. For example, using the following command you can let Jane download the siamese.jpg object and you can let everyone who is a member of the Google Cloud Storage group download the siamese.jpg object:

gsutil acl ch -g gs-discussion@googlegroups.com:R -u jane@gmail.com:R gs://cats/siamese.jpg

If you simply want to apply a predefined ACL (also known as a canned ACL) to a bucket or object, you can use the following command:

gsutil acl set bucket-owner-full-control gs://cats/siamese.jpg

To learn more about sharing data and modifying ACLs, see Access Control.

Step 7: Delete buckets and objects

You can delete objects with the rm command and you can delete buckets with the rb command. For example, the following command deletes the collie.jpg object that's in the cats bucket:

gsutil rm gs://cats/collie.jpg

The following command deletes the dogs bucket:

gsutil rb gs://dogs

You must remove all objects from a bucket before you delete it. You can do this by using the rm command or by using the mv command to move objects from one bucket to another, which you did when you moved the objects from the dogs bucket to the cats bucket.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.