Google Cloud Storage

Install gsutil

This pages discusses how to install gsutil, a tool that enables you to access Google Cloud Storage from the command-line.

Contents

System Requirements

gsutil runs on Linux/Unix, Mac OS, and Windows. To use gsutil, you must have Python 2.6.x or 2.7.x installed on your computer. gsutil does not currently run on Python 3.x. Python is installed by default on most distributions of Linux and Mac OS X. Python is not installed by default on Windows; you must install Python before you can run gsutil on Windows.

Warning: Please be aware that a tool also named gsutil, by GrandStream BudgeTone, is pre-installed on some Linux OS distributions. If you run this command instead of the Google Cloud Storage gsutil it will likely print an error message like "Choose one of -b, -d, -e, or -r to do something". If this happens, you can either move the Google Cloud Storage version of gsutil to the front of your PATH environment variable, or you can specify the full path when attempting to run Google Cloud Storage gsutil (e.g., /home/users/joan/gsutil/gsutil ls).

The following procedure helps you install gsutil on Windows so that you can run gsutil at the Windows command prompt. Although gsutil will run in this configuration, we recommend that you run gsutil in a CygWin environment if you want to run it on Windows.

How to Install gsutil

The following instructions help you install gsutil but does not discuss how to authenticate to the Google Cloud Storage service. You do not need to authenticate to the Google Cloud Storage service unless you want to access protected data, in which case, you should review the How to Set Up Credentials to Access Protected Data section below.

Installing from tar or zip Archive

This is the officially supported installation method for gsutil.

Linux and Mac OS X


  1. You can download gsutil as gsutil.tar.gz or as gsutil.zip. It doesn't matter which of these you use; however, these installation instructions assume you are using gsutil.tar.gz.

    Whichever format you download, gsutil is bundled in a single archive. Save the archive in a convenient location.

  2. Extract the archive files.

    Open a shell window, change directories to where you downloaded gsutil, and run the following command:

    tar xfz gsutil.tar.gz -C $HOME

    This installs gsutil in your home directory and creates a directory named gsutil. The gsutil directory should contain the gsutil tool, a gslib directory, and a third_party directory that contains several directories, including the boto library.

    If you want to install gsutil in a different location, replace $HOME with another location on your computer.

  3. Add gsutil to your PATH environment variable.

    Add the following line to your ~/.bashrc file on Linux or Windows with Cygwin, or to your ~/.bash_profile on Mac OS X:

    export PATH=${PATH}:$HOME/gsutil

    If you did not install gsutil in your home directory, be sure to change $HOME/gsutil so that it matches the location of gsutil.

  4. Restart your shell or terminal program.

    In some cases you might have to restart your shell or terminal program in order for the PATH environment variable to take effect.

That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Updating gsutil

Gsutil has a built-in update command that checks for the latest gsutil release and offers to let you automatically update to it if it differs from the version you have currently installed. Simply run the following command:

gsutil update

Windows


If you are using Windows with CygWin, follow the Linux installation instructions instead.

Note: The following procedure helps you install gsutil on Windows so that you can run gsutil at the Windows command prompt. Although gsutil will run in this configuration, we recommend that you run gsutil in a CygWin environment if you want to run it on Windows.

  1. Download and install Python.

    Install a version of Python that will work with gsutil (see System Requirements above). We recommend that you install Python in its default location (for example, C:\Python27).

  2. Download gsutil.

    gsutil is bundled in a single archive named gsutil.zip. Save the archive in a convenient location, such as C:\.

  3. Extract the archive files.

    We recommend that you extract the archive files into your root directory. This should create a directory named gsutil in your root directory (for example, C:\gsutil). The gsutil directory should contain the gsutil tool, a gslib directory, and a third_party directory that contains several directories, including the boto library.

  4. Start a command prompt window and run python gsutil to get a list of gsutil commands.

That's it. You're ready to start using gsutil.

Note: To run gsutil commands on Windows you must first call the Python interpreter. For example, to see a listing of gsutil commands you must type python gsutil at the command prompt. Also, if you run gsutil outside of the C:\gsutil directory you must provide the explicit path to the gsutil executable. For example, to run gsutil in the C:\Users\Admin\My Documents directory you must type python C:\gsutil\gsutil at the command prompt.

Updating gsutil

gsutil has a built-in update command that checks for the latest gsutil release and offers to let you automatically update to it if it differs from the version you have currently installed. Simply run the following command:

python gsutil update

Facilitating the Use of gsutil

The following steps show how to set environmental variables permanently in the Advanced System Properties. This can facilitate the use of the tool.

  1. Associate file names with .py extension to the Python interpreter.
  2. Rename gsutil to gsutil.py. This enables command line execution.
  3. Add the gsutil directory absolute path to the PATH environmental variable.
  4. Create or append gsutil\boto absolute path to the PYTHONPATH environment variable.
  5. If it does not exist, create a HOME environmental variable to locate the tool and set the proper access rights (try echo %USERPROFILE% in a command window).
  6. In a command line console execute gsutil.py ls to create the .boto config file. Any valid command will work.

Now you can use the python command to run gsutil from a command line console.

To learn how to use gsutil, see the Hello Google Cloud Storage! exercises, or simply run "gsutil help".

Installing from the Python Package Index (PyPi)

Experimental!

Installing gsutil via PyPi is an experimental and rapidly changing new feature for Google Cloud Storage. Unfortunately, this means that we may make backwards-incompatible changes. We will inform the community when this feature is no longer experimental.
 

CentOS, RHEL, and Fedora


  1. Install required system packages.

    Several packages are required to successfully install gsutil from PyPi. You can install them with the following command:

    sudo yum install gcc openssl-devel python-devel python-setuptools

  2. Install pip.

    We recommend using the pip installer. You can install it with the following command:

    sudo easy_install -U pip

  3. Install gsutil from PyPi.

    To install gsutil from PyPi, use the following command:

    sudo pip install gsutil

That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Updating gsutil

The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:

sudo pip install -U gsutil

Debian and Ubuntu


  1. Install required system packages.

    Several packages are required to successfully install gsutil from PyPi. You can install them with the following command:

    sudo apt-get install gcc python-dev python-setuptools

  2. Install pip.

    We recommend using the pip installer. You can install it with the following command:

    sudo easy_install -U pip

  3. Install gsutil from PyPi.

    To install gsutil from PyPi, use the following command:

    sudo pip install gsutil

That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Updating gsutil

The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:

sudo pip install -U gsutil

Mac OS X


  1. Install pip.

    We recommend using the pip installer. You can install it with the following command:

    sudo easy_install -U pip

  2. Install gsutil from PyPi.

    To install gsutil from PyPi, use the following command:

    sudo pip install gsutil

That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Updating gsutil

The gsutil update command is not available when installing via PyPi. To update gsutil, use the pip installer:

sudo pip install -U gsutil

Windows


Installing gsutil from PyPi is not support on Windows.

How to Set Up Credentials to Access Protected Data

Installing gsutil allows you to download and access publicly-accessible objects, but in order to access protected data or write to a protected bucket, you need to set up credentials to access the Google Cloud Storage service. For example, if someone else has created a Google Cloud Storage account and uploaded data that is only accessible to you or other specific individuals, you need to set up your credentials to the Google Cloud Storage service to be able to access this data.

Authenticate to Google Cloud Storage on Linux and OS X


  1. Open a new command prompt instance.
  2. Run gsutil config at the command prompt.

    The first time you run this command, gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:

    This script will create a boto config file at <your_homedir>/.boto
    containing your credentials, based on your responses to the following questions.
    
    Please navigate your browser to the following URL: <http://urlto/authorization/dialog>
    In your browser you should see a page that requests you to authorize gsutil to access
    Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed.
    
    Enter the authorization code:
    

    Note: If you want gsutil to launch a browser window for you, type gsutil config -b instead of gsutil config. However, if you are using SSH to connect to a remote machine, the -b option may not work.


  3. Copy and paste the URL into a browser window.

    If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.

  4. Click the Allow Access button.

    This authorizes gsutil to access Google Cloud Storage.

  5. Copy the authorization code that appears on the next page into the gsutil prompt and press Enter.
  6. Log in to the Google Developers Console to find a project ID you can specify as the default project.

    After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console. Copy and paste the URL into a browser window.

    Please navigate your browser to the following URL:
    https://console.developers.google.com/
    What is your project-id?
    
  7. Copy and paste the project ID into gsutil.

    Choose a project to set as the default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt. If you were given a project ID, you can enter it here. You can only set one project ID for gsutil at a time.

    You can change the project ID for gsutil at any time by editing the boto configuration file.

That's it. You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Authenticate to Google Cloud Storage on Windows


If you are using Windows with CygWin, follow the Linux authentication instructions above.

Note: The following procedure helps you authenticate gsutil on Windows. Although gsutil will run in this configuration, we recommend that you run gsutil in a CygWin environment if you want to run it on Windows.

  1. Start a command prompt window.
  2. Switch to the C:\gsutil directory.
  3. Run python gsutil config at the command prompt.

    The first time you run this command, gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:

    This script will create a boto config file at <your_homedir>/.boto
    containing your credentials, based on your responses to the following questions.
    
    Please navigate your browser to the following URL: <http://urlto/authorization/dialog>
    In your browser you should see a page that requests you to authorize gsutil to access
    Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed.
    
    Enter the authorization code:
    

    Note: If you want gsutil to launch a browser window for you, type python gsutil config -b instead of python gsutil config. However, if you are using SSH to connect to a remote machine, the -b option may not work.


  4. Copy and paste the URL into a browser window.

    If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.

  5. Click the Allow Access button.

    This authorizes gsutil to access Google Cloud Storage.

  6. Copy the authorization code that appears on the next page into the gsutil prompt.
  7. Log in to the Google Developers Console to find a project ID you can specify as the default project.

    After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console. Copy and paste the URL into a browser window.

    Please navigate your browser to the following URL:
    https://console.developers.google.com/
    What is your project-id?
    
  8. Copy and paste the project ID into gsutil.

    Choose a project to set as default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt. If you were given a project ID, you can enter it here. You can only set one project ID for gsutil at a time.

    You can change the project ID for gsutil at any time by editing the boto configuration file.

That's it. You're ready to start using gsutil.

Note: To run gsutil commands on Windows you must first call the Python interpreter. For example, to see a listing of gsutil commands you must type python gsutil at the command prompt. Also, if you run gsutil outside of the C:\gsutil directory you must provide the explicit path to the gsutil executable. For example, to run gsutil in the C:\Users\Admin\My Documents directory you must type python C:\gsutil\gsutil at the command prompt.

To learn how to use gsutil, see the Hello Google Cloud Storage! exercises.

How to Convert gsutil to Use OAuth 2.0

If you are currently using HMAC authentication and developer keys with gsutil, we recommend that you switch to using OAuth 2.0 authentication instead. Follow these instructions to set up OAuth 2.0 for gsutil.

  1. Download the latest version of gsutil.
  2. Extract the archive files to the directory where gsutil is currently installed.

    tar xfz gsutil.tar.gz -C $HOME

    If gsutil is currently installed in another directory, replace $HOME with that location. The new gsutil files will replace the existing files.

    For Windows, you will need to download an extraction tool to extract the archive files. The Windows operating system does not provide an extraction utility for .tar files.

  3. Run gsutil config at the command prompt.

    gsutil prints a URL to the authorization page where you can authorize gsutil to access Google Cloud Storage. The gsutil dialog looks similar to the following:

    Backing up existing config file '<your_homedir>/.boto' to '<your_homedir>/.boto.bak'...
    
    This script will create a boto config file at <your_homedir>/.boto
    containing your credentials, based on your responses to the following questions.
    
    Please navigate your browser to the following URL:  <http://urlto/authorization/dialog>
    In your browser you should see a page that requests you to authorize gsutil to access
    Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed.
    
    Enter the authorization code:
    

    Note: If you want gsutil to launch a browser window for you, type gsutil config -b instead of gsutil config. However, if you are using SSH to connect to a remote machine, the -b option may not work.

    If gsutil shows an error message like the following:

    CommandException: Cannot back up existing config file '<your_homedir>/.boto':
    backup file exists ('<your_homedir>.boto.bak').

    gsutil could not create a new configuration file because there are two existing configuration files. You will need to move or rename one file in order for gsutil to continue the configuration process.

    It is also possible to express a different scope for gsutil by including the following flags with the gsutil config command:

    • Request token with read-only access: -r
    • Request token with read-write access: -w
    • Request token with full-control (including changing Access Control Lists (ACLs): -f
    • Request a token with scope for other Google Services: -s <scope>

      Note: Generally, gsutil does not have the ability to make requests to other Google APIs. However, you can use this option in combination with the -r, -w, or -f flags to obtain a token that can access both Google Cloud Storage and another Google API. To find the specific scope strings for an API, refer to the API's documentation.

  4. Copy and paste the URL into a browser window.

    If prompted, sign in with the Google account associated with your Google Cloud Storage data in the browser window.

  5. Click the Allow Access button.

    This authorizes gsutil to access Google Cloud Storage.

  6. Copy the authorization code that appears on the next page into the gsutil prompt.
  7. Log in to the Google Developers Console to find a project ID you can specify as the default project.

    After you provide an authorization code, gsutil prints the prompt below and prints the URL to the Google Developers Console. Copy and paste the URL into a browser window.

    Please navigate your browser to the following URL:
    https://console.developers.google.com/
    What is your project-id?
    
  8. Copy and paste the project ID into gsutil.

    Choose a project to set as default for gsutil by clicking on the Overview tab on the left, copying the project ID and entering the project ID into the gsutil prompt.

    You can change the project ID for gsutil at any time by editing the boto configuration file. When you use the XML or JSON API, you must provide a project ID for certain operations (such as creating buckets). gsutil allows you to specify a default project ID in the boto configuration file, so you don't need to explicitly specify the project ID on operations when you work with the default project.

Congratulations, gsutil is now set up to use OAuth 2.0 authentication!

How to use boto with Google Cloud Storage

To use boto with Google Cloud Storage, we recommend that you configure boto to use OAuth 2.0 authentication:

Linux and Mac OS X

  1. Set up your boto configuration file to use OAuth 2.0.

    You can do so using gsutil and running the command gsutil config. If you don't have gsutil installed, see the gsutil installation instructions.

  2. In your boto configuration file, make sure that the https_validate_certificates flag in the [Boto] section is not set to False.

    This implements server certificate validation. For information about the location of the configuration file, see boto configuration file.

  3. Include the gsutil, boto, and third_party libraries in your PYTHONPATH.

    Add the following line to your .bashrc or .bash_profile.

    export PYTHONPATH=${PYTHONPATH}:$HOME/gsutil/third_party/boto:$HOME/gsutil

    If you installed gsutil in a directory other than your home directory, replace $HOME with the directory where your gsutil directory is located.

  4. Import the oauth2_plugin.

    Run the following command to import the oauth2_plugin:

    from gslib.third_party.oauth2_plugin import oauth2_plugin

That's it! You can now use boto with OAuth 2.0 authentication. The following example uses boto with OAuth 2.0 authentication to request the contents of a text file named rose.text:

from boto import storage_uri
from gslib.third_party.oauth2_plugin import oauth2_plugin
content = storage_uri('gs://pub/shakespeare/rose.txt').get_contents_as_string()
print(content)

Windows

  1. Set up your boto configuration file set up to use OAuth 2.0.

    You can do so using gsutil and running the command python gsutil config. If you don't have gsutil installed, see the gsutil installation instructions.

  2. In your boto configuration file, make sure that the https_validate_certificates flag in the [Boto] section is not set to False.

    This implements server certificate validation. For information about the location of the configuration file, see boto configuration file.

  3. Include the gsutil, boto, and third_party libraries in your PYTHONPATH.
    1. Open System properties in Control Panel, click the Advanced tab, and then click Environment Variables.
    2. Create a new variable named PYTHONPATH and assign it the following value: C:\gsutil\third_party\boto.
    3. Reboot for the environment variable change to take effect.
  4. Import the oauth2_plugin.

    Run the following command to import the oauth2_plugin:

    from gslib.third_party.oauth2_plugin import oauth2_plugin

Thats it! You can now use boto with OAuth 2.0 authentication. The following example uses boto with OAuth 2.0 authentication to request the contents of a text file named rose.text:

from boto import storage_uri
from gslib.third_party.oauth2_plugin import oauth2_plugin
content = storage_uri('gs://pub/shakespeare/rose.txt').get_contents_as_string()
print(content)

boto Configuration File

The boto/gsutil configuration file specifies values that control how gsutil behaves, as well as credentials and OAuth2 settings. You typically work with the configuration file indirectly by using the gsutil config command. In some situations, you might need to edit the file directly, for example, to change the default project ID. For more information about what the boto configuration file contains, see gsutil config.

The default location for the boto configuration file is in the user home directory, ~/.boto, for Linux and Mac OS X, and in %HOMEDRIVE%%HOMEPATH%, for Windows. You can get the location of the configuration file by running the command gsutil version -l.

You can override where the config file is used by setting the BOTO_CONFIG environment variable. You can also set up a path of boto config files to load. This could be useful, for example, in data sharing and collaboration scenarios.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.