Google Cloud Storage

Object Change Notification

Experimental!

This is an experimental release of Object Change Notification. As a result, it may be unstable, changed in backward-incompatible ways and is not recommended for production use. It is not subject to any SLA or deprecation policy, and might not move forward to General Availability.
 

Object Change Notification can be used to notify an application when an object is updated or added to a bucket.

Content

How Object Change Notification Works

A client application can send a request to watch for a bucket's change notification events in order to be notified about changes to a bucket's objects.

Completing a watch request creates a new notification channel. A notification channel is the means by which a notification message is sent to an application watching a bucket. Only one type of notification channel, a web hook, is currently supported.

After a notification channel is initiated, Google Cloud Storage notifies the application any time an object is added, updated, or removed from the bucket. For example, when you add a new picture to a bucket, an application could be notified to create a thumbnail.

The figure below shows an example data flow for an application that processes change notifications. Any application server that can receive HTTPS POST requests can be used to process change notifications.

Object Change Notification Components
Object Change Notification Components

Object Change Notification Details

Terminology

The following table contains a description of several terms used throughput the Object Change Notification documentation:

Term Description
Application URL The URL of your application. This is the address where notifications will be sent. Note that this must be an HTTPS URL; HTTP URLs are not permitted.
Channel Identifier The identifier for a notification channel. Must be unique within a particular bucket, i.e., if there are multiple notification channels for a single bucket, each notification channel must have a distinct channel identifier. This identifier will be sent to your application along with each notification message.
Resource Identifier An opaque identifier for the resource being watched. The resource identifier is required for stopping a notification channel. You can retrieve this identifier from the response to a watch request or from the body of notification event messages.
Client Token (optional) Client tokens can be used to validate notifications events. To do this, set a custom client token with your watch request. Notification messages will contain this token so you can verify that they are authentic.

Watching a Bucket

To start watching a bucket for change notification events, you can use the gsutil notification command:

gsutil notification watchbucket [-i ChannelId] [-t ClientToken] ApplicationUrl gs://BucketName

This will create a notification channel that sends notification events to the given application URL for the given bucket. The notification channel will include the custom client token and channel identifier if specified.

An example POST request generated by gsutil for watching a bucket:

POST /storage/v1beta2/b/BucketName/o/watch?alt=json HTTP/1.1
Host: www.googleapis.com
Content-Length: 200
User-Agent: google-api-python-client/1.0
Content-Type: application/json
Authorization: Bearer OAuthToken

{
  "token": "ClientToken",
  "type": "web_hook",
  "id": "ChannelId",
  "address": "ApplicationUrl"
}

Notification Authorization

When watching a bucket, the notification channel being created will be associated with the Google Developers Console project of the application initiating the API request. This means, for example, that if a user gives access to an installed application or web application via an OAuth2 flow, a notification channel created by the application will be associated with the application's project, not the project containing the bucket being watched.

There are two steps to configuring authorization in an object change notification scenario:

Creating a Service Account

Since a service account is associated with a project, using a service account to watch a bucket will create a notification channel in the service account's project.

To authorize a new service account:

  1. Go to the Google Developers Console.
  2. Select a project to which the service account will be associated.
  3. In the left sidebar, under APIs & auth, select Credentials.
  4. Click CREATE NEW CLIENT ID.
  5. In the Create Client ID window, choose Service account.

    Client ID for Service Account

  6. Click Create Client ID.
  7. A New Public/Private key pair generated window is displayed and the private key is downloaded automatically. Note the private key's password and click Okay, got it.

    Public/Private Key Pair

  8. Find the Service Account you just created, and note the generated Email address or keep the browser open when you configure gsutil.

    Service Account Details

    If you have existing Client IDs, the new Service account will be at the end of the list. You can always return to the Developers Console to generate a new key.

  9. Configure gsutil to use the email address and downloaded key.

    Use the gsutil config with the -e option, which prompts for service account credentials. The following excerpt shows an example of what configuring looks like. $HOME should be based on your home directory and you will enter the email address and key location appropriate for your situation.

    $ gsutil config -e
    This script will create a boto config file at $HOME/.boto
    containing your credentials, based on your responses to the following questions.
    
    What is your service account email address? 123412341234-abc123abc123@developer.gserviceaccount.com
    What is the full path to your private key file? $HOME/xyz345xyz345-privatekey.p12
    What is the password for your service key file? If you haven't  set
    one explicitly, leave this line blank.
    
    Your private key file is readable by people other than yourself.
    This is a security risk, since anyone with this information can use your service account.
    
    Would you like gsutil to change the file permissions for you? (y/N) y
    ...
    What is your project-id? your-project-id
    
    Boto config file "$HOME/.boto" created. If you need to use a proxy to access the
    Internet please see the instructions in that file.

Identifying a Domain to Receive Notifications

Watch requests will only succeed if the notification URL is a domain whitelisted by the notification channel's project.

To whitelist a domain:

  1. Verify that you own the domain using the Google Webmaster Tools verification process.
  2. Log in to the Google Developers Console.
  3. Make sure you have the correct project selected.
  4. In the left sidebar, click APIs & auth.
  5. Under APIs & auth, click Push.
  6. Click Add domains to bring up a dialog box where you can enter whitelisted domains: Notification Endpoints Dialog
  7. Click Add domains to save your changes.

    If you have verified ownership of the domains you added, your changes are made. If you have not verified ownership for one or more domains, an error message is displayed and your changes are not made.

Removing a Notification Channel

To stop a notification channel, you can use the gsutil notification command:

gsutil notification stopchannel ChannelId ResourceId

This will stop all notification events to the specified resource identifier and channel identifier pair. Additional active channels for the same resource will be unaffected. The resource and channel identifiers can be found in the response of a watch request or in the body of notification event messages.

An example POST request generated by gsutil for stopping a channel:

POST /storage/v1beta2/channels/stop HTTP/1.1
Host: www.googleapis.com
Content-Length: 200
User-Agent: google-api-python-client/1.0
Content-Type: application/json
Authorization: Bearer OAuthToken

{
  "resourceId": "ResourceId",
  "id": "ChannelId"
}

Notification Event Message Types

There are three types of notification event message types:

Sync

A notification event is sent when a new notification channel is created after issuing a watch request. After receiving the sync event, all later changes to the bucket will be sent to the application URL configured for the channel.

The notification will be sent as a POST request to the configured application URL. There is no body in the request. The sync notification metadata is contained in the request's headers. The following is an example of the sync notification request:

POST /ApplicationUrlPath
Accept: */*
Content-Type: application/json; charset="utf-8"
Content_Length: 0
Host: ApplicationUrlHost
User-Agent: APIs-Google; (+http://code.google.com/apis)
X-Goog-Channel-Id: ChannelId
X-Goog-Channel-Token: ClientToken
X-Goog-Message-Number: 1
X-Goog-Resource-Id: ResourceId
X-Goog-Resource-State: sync
X-Goog-Resource-Uri: https://www.googleapis.com/storage/v1beta2/b/BucketName/o?alt=json
Object Addition, Update, or Deletion

A notification event is sent when a new object is added to a bucket, an existing object's content or metadata has been modified, or an object is deleted from a bucket.

The notification will be sent as a POST request to the configured application URL. The body of the request contains a JSON-encoded message as shown in the following notification request:

POST /ApplicationUrlPath
Accept: */*
Content-Length: 1097
Content-Type: application/json; charset="utf-8"
Host: ApplicationUrlHost
User-Agent: APIs-Google; (+http://code.google.com/apis)
X-Goog-Channel-Id: ChannelId
X-Goog-Channel-Token: ClientToken
X-Goog-Message-Number: 16820
X-Goog-Resource-Id: ResourceId
X-Goog-Resource-State: ResourceState
X-Goog-Resource-Uri: https://www.googleapis.com/storage/v1beta2/b/BucketName/o?alt=json

{
 "kind": "storage#object",
 "id": "BucketName/ObjectName",
 "selfLink": "https://www.googleapis.com/storage/v1beta2/b/BucketName/o/ObjectName",
 "name": "ObjectName",
 "bucket": "BucketName",
 "generation": "1367014943964000",
 "metageneration": "1",
 "contentType": "binary/octet-stream",
 "updated": "2013-04-26T22:22:23.832Z",
 "size": "10",
 "md5Hash": "xHZY0QLVuYng2gnOQD90Yw==",
 "mediaLink": "https://www.googleapis.com/storage/v1beta2/b/BucketName/o/ObjectName?generation=1367014943964000&alt=media",
 "owner": {
  "entity": "user-007b2a38086590de0a47c786e54b1d0a21c02d062fcf3ebbaf9b63edb9c8db0c",
  "entityId": "007b2a38086590de0a47c786e54b1d0a21c02d062fcf3ebbaf9b63edb9c8db0c"
 },
 "crc32c": "C7+82w==",
 "etag": "COD2jMGv6bYCEAE="
}
where ResourceState is:
  • exists - for object additions and updates.
  • not_exists - for object deletions.

For a description of the contents of the JSON message, see the Object Resource Description.

Reliable Delivery

Object Change Notification will attempt to deliver notifications to your application in a reliable way. Since your application might not always be available, the following rules are followed:

  • If a notification delivery attempt fails, additional attempts will be made. The period of delay between delivery attempts is determined with an exponential backoff algorithm, starting with 30 seconds and increasing up to 90 minutes.
  • If your application responds with one of the following HTTP response codes, the notification delivery attempt is treated as a failure:
    • 401 Unauthorized
    • 404 Not Found
    • 500 Internal Server Error
    • 502 Bad Gateway
    • 503 Service Unavailable
    • 504 Gateway Timeout
  • If your application responds with one of the following HTTP response codes, the notification is treated as successfully delivered:
    • 102 Processing
    • 200 OK
    • 201 Created
    • 202 Accepted
    • 204 No Content
  • If your application cannot be reached after 20 seconds, the notification delivery attempt is treated as a failure.

Client Application Example

This section shows the steps you must perform to create an App Engine client application which processes change notification events.

The following figure shows the timeline of the steps involved to set up this example application, from creating a bucket to processing the related change notification events:

Object Change Notification Timeline
Object Change Notification Timeline

The example application contains a class called MainPage. When the user updates or adds an object to the bucket, the MainPage class processes the notification event. For simplicity, the post method that does the actual processing just logs a message with the time the notification was received. You can replace this code with your actual processing logic.

  1. Configuring the application.
    Create the configuration file app.yaml to specify the client application that handles the bucket's change notification events.
    application: <ApplicationId>
    version: 1
    runtime: python27
    api_version: 1
    threadsafe: true
    
    handlers:
    - url: /.*
      script: change_notification_client.app
    
  2. Assigning the application's access permission to the bucket.
    Grant to the application FULL_CONTROL access to the bucket by running the following command:
    gsutil acl ch -u ApplicationId@appspot.gserviceaccount.com:FC gs://BucketName
    
  3. Creating the Application.
    The following example implements a client application for handling a bucket's change notification events:
    """Notification handling for Google Cloud Storage."""
    
    import json
    import logging
    
    import webapp2
    
    
    class MainPage(webapp2.RequestHandler):
      """Process notification events."""
      def get(self):
        logging.info("Get request to notification page.")
        self.response.write("Welcome to the notification app.")
    
      def post(self):  # pylint: disable-msg=C6409
        """Process the notification event.
    
        This method is invoked when the notification channel is first created with
        a sync event, and then subsequently every time an object is added to the
        bucket, updated (both content and metadata) or removed. It records the
        notification message in the log.
        """
    
        logging.debug(
            '%s\n\n%s',
            '\n'.join(['%s: %s' % x for x in self.request.headers.iteritems()]),
            self.request.body)
    
        # The following code is for demonstration. Replace
        # it with your own notification processing code.
    
        if 'X-Goog-Resource-State' in self.request.headers:
          resource_state = self.request.headers['X-Goog-Resource-State']
          if resource_state == 'sync':
            logging.info('Sync message received.')
          else:
            an_object = json.loads(self.request.body)
            bucket = an_object['bucket']
            object_name = an_object['name']
            logging.info('%s/%s %s', bucket, object_name, resource_state)
        else:
          logging.info("Other post.")
    
    logging.getLogger().setLevel(logging.DEBUG)
    app = webapp2.WSGIApplication([('/', MainPage)], debug=True)
    
  4. Start watching the bucket for object changes.
    Create a notification channel to your application by watching the bucket with gsutil:
    gsutil notification watchbucket ApplicationUrl gs://BucketName
    
    where ApplicationUrl is the URL of your App Engine application, e.g. https://ApplicationId.appspot.com/.
  5. Testing the Application.
    To see if the application works as expected, perform the following steps:
    1. To assure that the application has been deployed and works correctly, execute the following curl command:
      curl -X Post https://<ApplicationId>.appspot.com
      
      If you used your own domain name to deploy the application, use it instead of appspot.com in the previous command.
    2. Go to the Logs section of your application in the App Engine dashboard. Refresh the list of the logged messages, if needed. Verify that the log message issued by the application is logged.
    3. Add an object to the bucket. You can use the gsutil tool as follows:
      gsutil cp <ObjectName> gs://<BucketName>/
      
      Google Cloud Storage notifies the application, which then logs a message.
    4. Go to the Logs section of your application in the App Engine dashboard. Refresh the list of the logged messages, if needed. You should be getting a message similar to this:

      Object Change Notification Message
      Object Change Notification Message

  6. Remove the notification channel
    Remove the notification channel by specifying the channel and resource identifiers returned when you issued the notification command to watch the bucket.
    gsutil notification stopchannel <channel_id> <resource_identifier>
    

Back to top

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.