Using Google Base and Google Gears for a Performant, Offline Experience

First article in the "Building Better Ajax Applications with Google APIs" series.

Dion Almaer and Pamela Fox, Google
June 2007

Editor's Note: Google Gears API is no longer available.

Introduction

Combining Google Base with Google Gears, we demonstrate how to create an application that can be used offline. After reading through this article, you will be more familiar with the Google Base API, as well as understand how to use Google Gears for storing and accessing user preferences and data.

Understanding the App

To understand this app, you should first be familiar with Google Base, which is basically a big database of items spanning various categories like products, reviews, recipes, events, and more.

Each item is annotated with a title, description, link to original source of the data (if exists), plus additional attributes that vary per category type. Google Base takes advantage of the fact that items in the same category share a common set of attributes-for example, all recipes have ingredients. Google Base items will even occasionally show up in search results from Google web search or Google product search.

Our demo app, Base with Gears, lets you store and display common searches you might perform on Google Base-like finding recipes with "chocolate" (yum) or personal ads with "walks on the beach" (romantic!). You can think of it as a "Google Base Reader" that lets you subscribe to searches and see the updated results when you revisit the app, or when the app goes out to look for updated feeds every 15 minutes.

Developers looking to extend the app could add more features, like visually alerting the user when the search results contain new results, letting the user bookmark (star) favorite items (offline + online), and letting the user do category-specific attribute searches like Google Base.

Using Google Base data API Feeds

Google Base can be queried programmatically with the Google Base data API, which is compliant with the Google Data API framework. The Google Data API protocol provides a simple protocol for reading and writing on the web, and is used by many Google products: Picasa, Spreadsheets, Blogger, Calendar, Notebook, and more.

The Google Data API format is based on XML and the Atom Publishing Protocol, so most of the read/write interactions are in XML.

An example of a Google Base feed based on the Google Data API is:
http://www.google.com/base/feeds/snippets/-/products?bq=digital+camera

The snippets feed type gives us the publicly available feed of items. The -/products lets us restrict the feed to the products category. And the bq= parameter lets us restrict the feed further, to only results containing the keyword "digital camera." If you view this feed in the browser, you'll see XML containing <entry> nodes with matching results. Each entry contains the typical author, title, content, and link elements, but also comes with additional category-specific attributes (like "price" for items in the products category).

Due to the cross-domain restriction of XMLHttpRequest in the browser, we aren't allowed to directly read in an XML feed from www.google.com in our JavaScript code. We could set up a server-side proxy to read in the XML and spit it back out at a location on the same domain as our app, but we would like to avoid server-side programming altogether. Luckily, there is an alternative.

Like the other Google Data APIs, the Google Base data API has a JSON output option, in addition to standard XML. The output for the feed we saw earlier in JSON format would be at this URL:
http://www.google.com/base/feeds/snippets/-/products?bq=digital+camera&alt=json

JSON is a lightweight interchange format that allows for hierarchical nesting as well as various data types. But more importantly, JSON output is native JavaScript code itself, and so it can be loaded into your web page just by referencing it in a script tag, bypassing the cross-domain restriction.

The Google Data APIs also let you specify a "json-in-script" output with a callback function to execute once the JSON is loaded. This makes the JSON output even easier to work with, as we can dynamically append script tags to the page and specify different callback functions for each.

So, to dynamically load a Base API JSON feed into the page, we could use the following function that creates a script tag with the feed URL (appended with altcallback values) and appends it to the page.

function getJSON() {
  var script = document.createElement('script');

  var url = "http://www.google.com/base/feeds/snippets/-/products?bq=digital+camera";
  script.setAttribute('src', url + "&alt=json-in-script&callback=listResults");
  script.setAttribute('type', 'text/JavaScript');
  document.documentElement.firstChild.appendChild(script);
}

So our callback function listResults can now iterate through the JSON-passed in as the only parameter-and display information on each entry found in a bulleted list. 

  function listTasks(root) {
    var feed = root.feed;
    var html = [''];
    html.push('<ul>');
    for (var i = 0; i < feed.entry.length; ++i) {
      var entry = feed.entry[i];
      var title = entry.title.$t;
      var content = entry.content.$t;
      html.push('<li>', title, ' (', content, ')</li>');
    }
    html.push('</ul>');

    document.getElementById("agenda").innerHTML = html.join("");
  }

Adding Google Gears

Now that we have an application that is able to talk to Google Base via the Google Data API, we want to enable this application to run offline. This is where Google Gears comes in.

There are various architecture choices when it comes to writing an application that can go offline. You will be asking yourself questions about how the application should work online vs. offline (e.g. Does it work exactly the same? Are some features disabled, such as search? How will you handle syncing?)

In our case, we wanted to make sure that the users on browsers without Gears can still use the app, while giving users who do have the plug-in the benefits of offline use and a more responsive UI.

Our architecture looks like this:

  • We have a JavaScript object that is in charge of storing your search queries and returning back results from these queries.
  • If you have Google Gears installed, you get a Gears version that stores everything in the local database.
  • If you do not have Google Gears installed, you get a version that stores the queries in a cookie and doesn't store the full results at all (hence the slightly slower responsiveness), as the results are too large to store in a cookie.
What is nice about this architecture is that you do not have to do checks for if (online) {} all over the shop. Instead, the application has one Gears check, and then the correct adapter is used.


Using a Gears Local Database

One of the components of Gears is the local SQLite database that is embedded and ready for your use. There is a simple database API that would look familiar to you if you have previously used APIs for server-side databases like MySQL or Oracle.

The steps to using a local database are quite simple:

  • Initialize the Google Gears objects
  • Get a database factory object, and open a database
  • Start executing SQL requests

Let's walk through these quickly.


Initialize the Google Gears Objects

Your application should read in the contents of /gears/samples/gears_init.js either directly, or by pasting in the code to your own JavaScript file. Once you have <script src="..../gears_init.js" type="text/JavaScript"></script> going, you have access to the google.gears namespace.


Get a Database Factory Object & Open a Database 
var db = google.gears.factory.create('beta.database', '1.0');
db.open('testdb');

This one call will give you a database object that allows you to open a database schema. When you open databases, they are scoped via the same origin policy rules, so your "testdb" won't clash with my "testdb."


Start Executing SQL Requests

Now we are ready to send SQL requests to the database. When we send "select" requests, we get back a result set that we can iterate through for desired data: 

var rs = db.execute('select * from foo where name = ?', [ name ]);

You can operate on the returned result set with the following methods:

booleanisValidRow()
voidnext()
voidclose()
intfieldCount()
stringfieldName(int fieldIndex)
variantfield(int fieldIndex)
variantfieldByName(string fieldname)

For more details, please see the Database Module API documentation. (Editor's Note: Google Gears API is no longer available).


Using GearsDB to Encapsulate the Low Level API

We wanted to encapsulate and make more convenient some of the common database tasks. For example,

  • We wanted to have a nice way to log the SQL that was generated when we were debugging the application.
  • We wanted to handle exceptions in one place instead of having to try{}catch(){} all over the place.
  • We wanted to deal with JavaScript objects instead of result sets when reading or writing data.

To handle these issues in a generic way, we created GearsDB, an open source library that wraps the Database object. We will now show how to make use of GearsDB.

Initial Setup

In our window.onload code, we need to make sure that the database tables that we rely on are in place. If the user has Gears installed when the following code runs, they will create a GearsBaseContent object: 

content = hasGears() ? new GearsBaseContent() : new CookieBaseContent();

Next, we open the database and create tables if they don't already exist: 

db = new GearsDB('gears-base'); // db is defined as a global for reuse later!

if (db) {
  db.run('create table if not exists BaseQueries' +
         ' (Phrase varchar(255), Itemtype varchar(100))');
  db.run('create table if not exists BaseFeeds' + 
         ' (id varchar(255), JSON text)');
}

At this point, we are sure that we have a table to store the queries and the feeds. The code new GearsDB(name) will encapsulate the opening of a database with the given name. The run method wraps the lower level execute method but also handles debugging output to a console and trapping exceptions.


Adding a Search Term

When you first run the app, you won't have any searches. If you try to search for a Nintendo Wii in products, we will save this search term in the BaseQueries table.

The Gears version of the addQuery method does this by taking the input and saving it via insertRow: 

var searchterm = { Phrase: phrase, Itemtype: itemtype };
db.insertRow('BaseQueries', searchterm);

insertRow takes a JavaScript object (searchterm) and handles INSERTing it into the table for you. It also lets you define constraints (for example, uniqueness-block insertion of more than one "Bob"). However, most of the time you will be handling these constraints in the database itself.


Getting All Search Terms

To populate your list of past searches, we use a nice select wrapper named selectAll: 

GearsBaseContent.prototype.getQueries = function() {
  return this.db.selectAll('select * from BaseQueries');
}

This will return an array of JavaScript objects that match the rows in the database (e.g. [ { Phrase: 'Nintendo Wii', Itemtype: 'product' }, { ... }, ...]).

In this case, it's fine to return the full list. But if you have a lot of data, you'll probably want to use a callback in the select call so that you can operate on each returned row as it comes in: 

 db.selectAll('select * from BaseQueries where Itemtype = ?', ['product'], function(row) {
  ... do something with this row ...
});

Here are some other helpful select methods in GearsDB:

selectOne(sql, args)Return first/one matching JavaScript object
selectRow(table, where, args, select)Normally used in simple cases to ignore SQL
selectRows(table, where, args, callback, select)Same as selectRow, but for multiple results.

Loading a Feed

When we get the results feed from Google Base, we need to save it to the database: 

content.setFeed({ id: id, JSON: json.toJSONString() });

... which calls ...

GearsBaseContent.prototype.setFeed = function(feed) {
  this.db.forceRow('BaseFeeds', feed);
}

We first take the JSON feed and return it as a String using the toJSONString method. Then we create the feed object and pass that into the forceRow method. forceRow will INSERT an entry if one doesn't already exist, or UPDATE an existing entry.


Displaying Search Results

Our app displays the results for a given search on the right panel of the page. Here's how we retrieve the feed associated with the search term: 

GearsBaseContent.prototype.getFeed = function(url) {
  var row = this.db.selectRow('BaseFeeds', 'id = ?', [ url ]);
  return row.JSON;
}

Now that we have the JSON for a row, we can eval() it to get the objects back: 

eval("var json = " + jsonString + ";");

We are off to the races and can start innerHTMLing content from JSON into our page.


Using a Resource Store for Offline Access

Since we are getting content from a local database, this app should also work offline, right?

Well, no. The problem is that to kick off this app, you need to load its web resources, such as its JavaScript, CSS, HTML, and images. As it currently stands, if your user took the following steps, the app might still work: start online, do some searches, don't close browser, go offline. This could work as the items would still be in the browser's cache. But what if this isn't the case? We want our users to be able to access the app from scratch, after a reboot, etc.

To do this, we use the LocalServer component and capture our resources. When you capture a resource (such as the HTML and JavaScript required to run the application), Gears will save away these items and will also trap requests from the browser to return them. The local server will act as a traffic cop and return the saved contents from the store.

We also make use of the ResourceStore component, which requires you to manually tell the system which files you want to capture. In many scenarios, you want to version your application and allow for upgrades in a transactional way. A set of resources together define a version, and when you release a new set of resources you will want your users to have a seamless upgrade of the files. If that's your model, then you will be using the ManagedResourceStore API.

To capture our resources, the GearsBaseContent object will:

  1. Set up an array of files that needs capturing
  2. Create a LocalServer
  3. Open or create a new ResourceStore
  4. Call out to capture the pages into the store
// Step 1
this.storeName = 'gears-base';
this.pageFiles = [
  location.pathname,
  'gears_base.js',
  '../scripts/gears_db.js',
  '../scripts/firebug/firebug.js',
  '../scripts/firebug/firebug.html',
  '../scripts/firebug/firebug.css',
  '../scripts/json_util.js',    'style.css',
  'capture.gif' ];

// Step 2
try {
  this.localServer = google.gears.factory.create('beta.localserver', '1.0');
} catch (e) {
  alert('Could not create local server: ' + e.message);
  return;
}

// Step 3
this.store = this.localServer.openStore(this.storeName) || this.localServer.createStore(this.storeName);

// Step 4
this.capturePageFiles();

... which calls ...

GearsBaseContent.prototype.capturePageFiles = function() {
  this.store.capture(this.pageFiles, function(url, success, captureId) {
    console.log(url + ' capture ' + (success ? 'succeeded' : 'failed'));
  });
}

What is important to note here is that you can only capture resources on your own domain. We ran into this limitation when we tried to access the GearsDB JavaScript file directly from the original "gears_db.js" file in its SVN trunk. The solution is simple, of course: you need to download any external resources and place them under your domain. Note that 302 or 301 redirects will not work, as LocalServer only accepts 200 (Success) or 304 (Not Modified) server codes.

This has implications. If you place your images on images.yourdomain.com, you will not be able to capture them. www1 and www2 can't see each other. You could set up server-side proxies, but that would defeat the purpose of splitting out your application to multiple domains.

Debugging the Offline Application

Debugging an offline application is a little more complicated. There are now more scenarios to test:

  • I am online with the app fully running in cache
  • I am online but have not accessed the app, and nothing in cache
  • I am offline but have accessed the app
  • I am offline and have never accessed the app (not a good place to be!)

To make life easier, we used the following pattern:

  • We disable the cache in Firefox (or your browser of choice) when we need to make sure that the browser isn't just picking something up from the cache
  • We debug using Firebug (and Firebug Lite for testing on other browsers); we use console.log() all over the place, and detect for the console just in case
  • We add helper JavaScript code to:
    • allow us to clear out the database and give us a clean slate
    • remove the captured files, so when you reload, it goes out to the Internet to get them again (useful when you are iterating on development ;)

The debug widget shows up on the left side of the page only if you have Gears installed. It has callouts to clean-up code: 

GearsBaseContent.prototype.clearServer = function() {
  if (this.localServer.openStore(this.storeName)) {
    this.localServer.removeStore(this.storeName);
    this.store = null;
  }
}

GearsBaseContent.prototype.clearTables = function() {
  if (this.db) {
    this.db.run('delete from BaseQueries');
    this.db.run('delete from BaseFeeds');
  }
  displayQueries();
}

Conclusion

You can see that working with Google Gears is actually fairly simple. We used GearsDB to make the Database component even easier, and used the manual ResourceStore, which worked just fine for our example.

The area where you spend the most time is defining the strategy for when to get data online, and how to store it offline. It is important to spend time on defining the database schema. If you do need to change the schema in the future, you will need to manage that change since your current users will have a version of the database already. This means that you will need to ship script code with any db upgrade. Obviously, it helps to minimize this, and you may want to try out GearShift, a small library that may help you manage revisions.

We could have also used ManagedResourceStore to keep track of our files, with the following consequences:

  • We would be good citizens and version our files to enable clean future upgrades
  • There is a feature of the ManagedResourceStore that lets you alias a URL to another piece of content. A valid architecture choice would be to have gears_base.js be a non-Gears version, and alias that so Gears itself would download gears_base_withgears.js which would have all of the offline support.
For our app, we felt it was easier just to have one interface and implement that interface in two ways.

We hope you found Gearing up applications fun and easy! Please join us in the Google Gears forum if you have questions or an app to share.