Google Developers Academy

Retrieving data from the Datastore

Learning objectives
  • Learn how to build and deploy an App Engine app, a simple guestbook

Amy Unruh, Dan Sanderson, Oct 2012
Google Developer Relations


The previous lesson in this class showed how to store guestbook data in the Datastore using the NDB API. In this lesson, we'll show how to access that stored data to display a log of guestbook entries.

Retrieving the Stored Greetings

The App Engine Datastore has a sophisticated query engine, which you can access via NDB. From the version of helloworld/ introduced in the previous lesson, the following new version of the MainPage handler queries the Datastore for greetings:

class MainPage(webapp2.RequestHandler):
  def get(self):
    guestbook_name = self.request.get('guestbook_name')
    # There is no need to actually create the parent Book entity; we can
    # set it to be the parent of another entity without explicitly creating it
    ancestor_key = ndb.Key("Book", guestbook_name or "*notitle*")
    greetings = Greeting.query_book(ancestor_key).fetch(20)

    for greeting in greetings:
            '<b>%s</b> wrote:' %
        self.response.out.write('An anonymous person wrote:')
      self.response.out.write('<blockquote>%s</blockquote>' %

          <form action="/sign" method="post">
            <input type="hidden" value="%s" name="guestbook_name">
            <div><textarea name="content" rows="3" cols="60"></textarea></div>
            <div><input type="submit" value="Sign Guestbook"></div>
          <form>Guestbook name: <input value="%s" name="guestbook_name">
          <input type="submit" value="switch"></form>
      </html>""" % (cgi.escape(guestbook_name), cgi.escape(guestbook_name)))

The query happens here, with the third line the one that actually executes the query:

    guestbook_name = self.request.get('guestbook_name')
    # There is no need to actually create the parent Book entity; we can
    # set it to be the parent of another entity without explicitly creating it
    ancestor_key = ndb.Key("Book", guestbook_name or "*notitle*")
    greetings = Greeting.query_book(ancestor_key).fetch(20)

We construct the greeting's ancestor key from the parent entity kind (Book) and the guestbook name. Then we use the ancestor key to query for greetings that have that ancestor, ordered by date (as defined in the query_book method). We're retrieving the first 20 of such greetings.

More on Data Consistency

Google App Engine's High Replication Datastore (HRD) provides high availability for your reads and writes by storing data synchronously in multiple data centers. However, the delay from the time a write is committed until it becomes visible in all data centers means that queries across multiple entity groups (non-ancestor queries) can only guarantee eventually consistent results. Consequently, the results of such queries may sometimes fail to reflect recent changes to the underlying data. However, a direct fetch of an entity by its key is always consistent.

In the example for this course, we are structuring our greeting data so that all the greetings for a given guestbook share the same ancestor entity, and thus belong to the same entity group. These means that queries for a guestbook's greetings will be strongly consistent, always reflecting all changes to the data right away. That is, because entity groups are a unit of consistency as well as transactionality, all data operations are applied to the entire group; an ancestor query won't return its results until the entire entity group is up to date. The supported write rate for entity groups is about one write per second.

If you don't constrain your Datastore entities to be in the same entity group, then queries for entities will be eventually consistent. This means that nearly all writes will be available for queries within a few seconds, but at times there might be a brief inconsistency. However, because your greeting entities don't belong to the same entity group, your write rate won't be limited to only one write per second per entity group. And a direct entity fetch by key is always consistent. Additionally, by using services such as Memcache, you can mitigate the chance that a user won't see fresh results when querying across entity groups immediately after a write. So in many contexts, eventual consistency can be an acceptable tradeoff.

Datastore Indexes

Every query in the App Engine Datastore is computed from one or more indexes, tables that map ordered property values to entity keys. This is how App Engine is able to serve results quickly regardless of the size of your application's Datastore.

App Engine automatically predefines an index for each property of each entity kind. These predefined indexes are sufficient to perform many simple queries, but for more complex queries the application must define the indexes it needs in an index configuration file named index.yaml. Without a custom index, the Datastore can't execute the query efficiently.

Our guestbook example above orders Greeting entities by descending date and uses an ancestor query and a sort order. This query requires a custom index to be specified in the application's index configuration file. It should look like the following:

- kind: Greeting
  ancestor: yes
  - name: date
    direction: desc

When you upload your application, the custom index definition in index.yaml will automatically be uploaded along with it.

Note: Exercising Datastore queries in your application locally causes App Engine to create or update index.yaml. If the file is missing or incomplete, you'll see index errors when your uploaded application executes queries for which the necessary indexes have not been specified. To avoid missing index errors in production, always test new queries at least once locally before uploading your application. See Python Datastore Index Configuration for more information.

You can read more about Datastore indexes on the Datastore Indexes documentation page. You can read about the proper specification for your index.yaml file on the Python Datastore Index Configuration page.

Review and Summary

In this lesson, we've shown how to query the Datastore, and introduced Datastore indexes and consistency concepts. We've barely scratched the surface of the Datastore and NDB's capabilities; see the documentation for more.

We now have a working guestbook application that lets users submit messages and displays messages that other users have left. However, this version mixes hard-wired HTML content with the code for the MainPage handler. This will make it difficult to change the appearance of the application, especially as our application gets bigger and more complex. In the next lesson, we'll use templates to improve this design.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.