Google App Engine

Using the XMPP service

Nick Johnson
September 2009
, updated January 2013

Introduction

With the introduction of the XMPP service to App Engine, it's now possible to write an App Engine app that communicates with users - or even other applications - over XMPP. XMPP is an instant-messaging protocol, used by Google Talk, Jabber, and other IM networks. In this article, we're going to walk through an example that covers all the basic functionality of App Engine's XMPP API.

For our example app, we're going to write the Amazing Crowd Guru. The Amazing Crowd Guru is a veritable oracle, who can answer any question you might pose it over XMPP. Writing an omniscient computer program is no small task, but thanks to a little behind-the-scenes trickery, we're going to get our users to do all the work of answering questions for us.

The basic sequence of events will go like this:

  1. A user adds crowdguru@appspot.com to their buddy list in Google Talk, or another XMPP client.
  2. The user asks the Amazing Crowd Guru a question, by typing "/tellme Does a duck's quack echo?"
  3. Our code receives the question, stores it in the datastore as an unanswered question, then looks in the datastore for another unanswered question. If it finds one, it sends it back to the user, saying "While I'm thinking, perhaps you can answer me this: If a mole can dig a mole of holes, how many moles of holes can a mole of moles dig?"
  4. The user thinks a bit, and replies "A mole of moles can dig a mole of holes!"
  5. Our code receives the user's answer, stores it in the datastore alongside the original question, and then sends it back to the user who originally asked that question.

In addition to the basic flow above, we'll add a couple of enhancements: A user can type "/help" to ask the Guru for a quick overview of what they can type, and they can type "/askme", to be asked a question by the Guru, without having to ask one of their own. We'll also suspend questions for users who are offline, to ensure that answers are delivered to users who are available.

Getting Started

First, we need to set up a few basic settings. Create a directory for your app, and create your configuration file:

Python

app.yaml
application: crowdguru-hrd
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /static
  static_dir: static

- url: /.*
  script: guru.APPLICATION

inbound_services:
- xmpp_message
- xmpp_presence

libraries:
- name: jinja2
  version: latest

Java

war/WEB-INF/appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>crowdguru</application>
  <version>1</version>
  <threadsafe>true</threadsafe>

  <system-properties>
    <property name="java.util.logging.config.file" value="WEB-INF/logging.properties" />
  </system-properties>

  <inbound-services>
    <service>xmpp_message</service>
    <service>xmpp_presence</service>
  </inbound-services>
</appengine-web-app>

If you're planning to deploy your version of the app to App Engine, you'll want to replace crowdguru with another app ID that you've created.

Next, if you're building your application in Python, create a handler script called guru.py and start it out with the necessary imports:

Python

guru.py
import datetime

from google.appengine.api import datastore_types
from google.appengine.api import xmpp
from google.appengine.ext import ndb
from google.appengine.ext.webapp import xmpp_handlers
import webapp2
from webapp2_extras import jinja2

All of this should seem very familiar from writing previous apps, so we won't go into much detail describing it. Note that we haven't defined any models or handlers yet - we'll do that next.

Creating a Model

Before we can do anything useful, we'll need a datastore model class to store questions and answers in. If you're building your application in Python, add the following to guru.py immediately after the import statements. If you're building in Java, create a new domain class, Question.java, with the following:

Python

guru.py
class Question(ndb.Model):
    """Model to hold questions that the Guru can answer."""
    question = ndb.TextProperty(required=True)
    asker = IMProperty(required=True)
    asked = ndb.DateTimeProperty(required=True, auto_now_add=True)
    suspended = ndb.BooleanProperty(required=True)

    assignees = IMProperty(repeated=True)
    last_assigned = ndb.DateTimeProperty()

    answer = ndb.TextProperty(indexed=True)
    answerer = IMProperty()
    answered = ndb.DateTimeProperty()

Note: The property class IMProperty is defined in the source for this application.

Java

src/Question.java
// Java JDO
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.Text;

import java.util.Date;

import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Question {

  @PrimaryKey
  @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
  private Key key;

  @Persistent(defaultFetchGroup = "true")
  private Text question;

  @Persistent
  private String asker;

  @Persistent
  private Date asked;

  @Persistent(defaultFetchGroup = "true")
  private Text answer;

  @Persistent
  private String answerer;

  @Persistent
  private Date answered;

  @Persistent
  private Boolean suspended;

  public Key getKey() {
    return key;
  }

  // ...

We've omitted a couple of helper methods here to save space, including several Python functions dealing with assigning questions to a user in a transaction-safe manner. If you want to see all the gory details, the source is freely available and linked from here.

Responding to XMPP messages

Finally, we're ready to tackle the meat of the problem: Receiving and responding to XMPP messages sent by users. As described in the introduction, we're using a pattern fairly common to 'bots': Users send us either plain text messages, or commands prefixed with a "/" character. Messages are interpreted based on the command specified and the (optional) argument.

The Python environment includes some convenience classes to make writing XMPP bots easier, namely CommandHandler in google.appengine.ext.webapp.xmpp_handlers. This class implements a "command dispatch" pattern: messages prefixed with "/foo" are handled by a method named foo_command, messages without a prefix are handled by text_message, and any messages with unknown prefixes are handled by unhandled_command which sends a message back to the sender saying "Unknown command".

While the Java environment does not have a special handler for dispatching XMPP commands, its XMPP API includes a handy parseMessage method for creating a new Message object from HTTP request data, and from there it's straightforward to inspect the body of the message to determine how to handle the command.

With these techniques, we can write the code to handle incoming requests to the Guru and send responses fairly simply:

Python

guru.py
def bare_jid(sender):
    return sender.split('/')[0]


class XmppHandler(xmpp_handlers.CommandHandler):
    """Handler class for all XMPP activity."""

    def tellme_command(self, message=None):
        """Handles /tellme requests, asking the Guru a question.

        Args:
            message: xmpp.Message: The message that was sent by the user.
        """
        im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
        asked_question = Question.get_asked(im_from)

        if asked_question:
            # Already have a question
            message.reply(WAIT_MSG)
        else:
            # Asking a question
            asked_question = Question(question=message.arg, asker=im_from)
            asked_question.put()

            currently_answering = Question.get_answering(im_from)
            if not currently_answering:
                # Try and find one for them to answer
                question = Question.assign_question(im_from)
                if question:
                    message.reply(TELLME_MSG.format(question.question))
                    return
            message.reply(PONDER_MSG)

Java

src/XmppReceiverServlet.java
import com.google.appengine.api.xmpp.JID;
import com.google.appengine.api.xmpp.Message;
import com.google.appengine.api.xmpp.MessageBuilder;
import com.google.appengine.api.xmpp.MessageType;
import com.google.appengine.api.xmpp.XMPPService;
import com.google.appengine.api.xmpp.XMPPServiceFactory;

import java.io.IOException;
import java.util.Date;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * Handler class for all XMPP messages.
 */
public class XmppReceiverServlet extends HttpServlet {

  private static final XMPPService xmppService =
      XMPPServiceFactory.getXMPPService();

  @Override
  public void doPost(HttpServletRequest req, HttpServletResponse resp)
      throws IOException {
    Message message = xmppService.parseMessage(req);

    if (message.getBody().startsWith("/askme")) {
      handleAskMeCommand(message);
    } else if (message.getBody().startsWith("/tellme ")) {
      String questionText = message.getBody().replaceFirst("/tellme ", "");
      handleTellMeCommand(message, questionText);
    } else if (message.getBody().startsWith("/")) {
      handleUnrecognizedCommand(message);
    } else {
      handleAnswer(message);
    }
  }

  /**
   * Handles /tellme requests, asking the Guru a question.
   */
  private void handleTellMeCommand(Message message, String questionText) {
    QuestionService questionService = new QuestionService();

    JID sender = message.getFromJid();
    Question previouslyAsked = questionService.getAsked(sender);

    if (previouslyAsked != null) {
      // Already have a question, and they're not answering one
      replyToMessage(message, "Please! One question at a time! You can ask " +
          "me another once you have an answer to your current question.");
    } else {
      // Asking a question
      Question question = new Question();
      question.setQuestion(questionText);
      question.setAsked(new Date());
      question.setAsker(sender);

      questionService.storeQuestion(question);

      // Try and find one for them to answer
      Question assigned = questionService.assignQuestion(sender);

      if (assigned != null) {
        replyToMessage(message, "While I'm thinking, perhaps you can " +
            "answer me this: " + assigned.getQuestion());
      } else {
        replyToMessage(message, "Hmm. Let me think on that a bit.");
      }
    }
  }

  // ...

  private void replyToMessage(Message message, String body) {
    Message reply = new MessageBuilder()
        .withRecipientJids(message.getFromJid())
        .withMessageType(MessageType.NORMAL)
        .withBody(body)
        .build();

    xmppService.sendMessage(reply);
  }
}

This is the core functionality: how users ask the Guru a question. We've left out the get_asked() (Python) and getAsked (Java) methods here for brevity; they are fairly straightforward methods that query the datastore for the question the user asked (if any). If you're interested, you can see the full, unmodified code, linked from here.

As you can see, the actual functionality of using XMPP is very straightforward. In Python, received messages result in a call to the appropriate function which is passed a message object. In Java, the message object is parsed directly from the posted request. The message object encapsulates information about the sender of the message, the JID (Jabber ID) it was sent to, and the body of the message. The Python implementation also provides convenience methods to parse the body of the message into command and argument portions (if it is of that form), and to reply to the message.

In this case, the only user we're sending messages to is the sender of the message we're handling, so everything is very straightforward - we just call message.reply() with the text we want to send back to the user.

Next, we need a way for the user to send us their answer to a question. They do this by simply typing the answer, with no /command prefix:

Python

guru.py
def text_message(self, message=None):
    """Called when a message not prefixed by a /cmd is sent to the XMPP bot.

    Args:
        message: xmpp.Message: The message that was sent by the user.
    """
    im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
    question = Question.get_answering(im_from)
    if question:
        other_assignees = question.assignees
        other_assignees.remove(im_from)

        # Answering a question
        question.answer = message.arg
        question.answerer = im_from
        question.assignees = []
        question.answered = datetime.datetime.now()
        question.put()

        # Send the answer to the asker
        xmpp.send_message([question.asker.address],
                          ANSWER_INTRO_MSG.format(question.question))
        xmpp.send_message([question.asker.address],
                          ANSWER_MSG.format(message.arg))

        # Send acknowledgement to the answerer
        asked_question = Question.get_asked(im_from)
        if asked_question:
            message.reply(TELLME_THANKS_MSG)
        else:
            message.reply(THANKS_MSG)

        # Tell any other assignees their help is no longer required
        if other_assignees:
            xmpp.send_message([user.address for user in other_assignees],
                              SOMEONE_ANSWERED_MSG)
    else:
        self.unhandled_command(message)

Java

src/XmppReceiverServlet.java
  // ...

  /**
   * Handles answers to questions we've asked the user.
   */
  private void handleAnswer(Message message) {
    QuestionService questionService = new QuestionService();

    JID sender = message.getFromJid();
    Question currentlyAnswering = questionService.getAnswering(sender);

    if (currentlyAnswering != null) {
      // Answering a question
      currentlyAnswering.setAnswer(message.getBody());
      currentlyAnswering.setAnswered(new Date());
      currentlyAnswering.setAnswerer(sender);

      questionService.storeQuestion(currentlyAnswering);

      // Send the answer to the asker
      sendMessage(currentlyAnswering.getAsker(), "I have thought long and " +
          "hard, and concluded: " + currentlyAnswering.getAnswer());

      // Send acknowledgment to the answerer
      Question previouslyAsked = questionService.getAsked(sender);

      if (previouslyAsked != null) {
        replyToMessage(message, "Thank you for your wisdom. I'm still " +
            "thinking about your question.");
      } else {
        replyToMessage(message, "Thank you for your wisdom.");
      }
    } else {
      handleUnrecognizedCommand(message);
    }
  }

  // ...

  private void sendMessage(String recipient, String body) {
    Message message = new MessageBuilder()
        .withRecipientJids(new JID(recipient))
        .withMessageType(MessageType.NORMAL)
        .withBody(body)
        .build();

    xmppService.sendMessage(message);
  }

This is a little longer than tellme_command, but still fairly easy to follow. First we look up the question the user is answering (again, we've omitted the get_answering (Python) and getAnswering (Java) methods for brevity). If they're not answering a question, we call unhandled_command() (Python) or handleUnrecognizedCommand (Java), which prints help text. Otherwise, we record the answer, send it back to the person who originally asked the question, and send an acknowledgement to the answerer. Note the bit where we send the answer to the original asker: This demonstrates using xmpp.send_message (Python) to send a message to a JID other than the sender of the message we're currently handling. In Java, we've added a custom method for sending a message to an arbitrary sender.

If a user asks a question and then goes offline, we shouldn't bother answering their question until they're around to hear the answer. We can track this using presence handlers.

Python

guru.py
class XmppPresenceHandler(webapp2.RequestHandler):
    """Handler class for XMPP status updates."""

    def post(self, status):
        """POST handler for XMPP presence.

        Args:
            status: A string which will be either available or unavailable
               and will indicate the status of the user.
        """
        sender = self.request.get('from')
        im_from = datastore_types.IM('xmpp', bare_jid(sender))
        suspend = (status == 'unavailable')
        query = Question.query(Question.asker == im_from,
                               Question.answer == None,
                               Question.suspended == (not suspend))
        question = query.get()
        if question:
            question.suspended = suspend
            question.put()

Java

  @Override
  public void doPost(HttpServletRequest request, HttpServletResponse response)
      throws IOException {
    Presence presence = xmppService.parsePresence(request);
    JID sender = presence.getFromJid();

    QuestionService questionService = new QuestionService();
    if (presence.getPresenceType() == PresenceType.AVAILABLE) {
      questionService.setSuspended(sender, true);
    } else if (presence.getPresenceType() == PresenceType.UNAVAILABLE) {
      questionService.setSuspended(sender, false);
    }
  }

When we are notified that a user has gone offline, we search our database for an unanswered question that they have asked. If one exists, we mark it as "suspended" and don't give it to other users to answer. When the user comes back online, we un-suspend the question so it's available to be answered again.

Finally, it would be nice if users have a way to answer questions without having to ask one first; this way, dedicated users can help clear out any backlog of unanswered questions:

Python

guru.py
def askme_command(self, message=None):
    """Responds to the /askme command.

    Args:
        message: xmpp.Message: The message that was sent by the user.
    """
    im_from = datastore_types.IM('xmpp', bare_jid(message.sender))
    currently_answering = Question.get_answering(im_from)
    question = Question.assign_question(im_from)
    if question:
        message.reply(TELLME_MSG.format(question.question))
    else:
        message.reply(EMPTYQ_MSG)
    # Don't unassign their current question until we've picked a new one.
    if currently_answering:
        currently_answering.unassign(im_from)

Java

src/XmppReceiverServlet.java
  // ...

  /**
   *  Handles requests to be asked a question without providing one.
   */
  private void handleAskMeCommand(Message message) {
    QuestionService questionService = new QuestionService();

    JID sender = message.getFromJid();
    Question newlyAssigned = questionService.assignQuestion(sender);

    if (newlyAssigned != null) {
      replyToMessage(message, "While I'm thinking, perhaps you can answer " +
          "me this: " + newlyAssigned.getQuestion());
    } else {
      replyToMessage(message, "Sorry, I don't have anything to ask you at " +
          "the moment.");
    }
  }

  // ...

This method is even simpler - grab a question and send it back to the user. In fact, it's pretty much just a subset of the code we saw in the tellme_command() method.

There's one last thing we need to do to get this all working, of course - hook it up to the serving infrastructure so it can serve requests. In Python, a CommandHandler is a standard webapp RequestHandler subclass, so we can set it up as we would any other handler. In Java, you can wire up the XMPP receiving servlet in your deployment descriptor.

Python

guru.py
APPLICATION = webapp2.WSGIApplication([
        ('/_ah/xmpp/message/chat/', XmppHandler),
        ('/_ah/xmpp/presence/(available|unavailable)/', XmppPresenceHandler),
        ], debug=True)

Java

war/WEB-INF/web.xml
<?xml version="1.0" encoding="utf-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  version="2.5">
  <servlet>
    <servlet-name>xmppreceiver</servlet-name>
    <servlet-class>com.google.appengine.demos.crowdguru.web.XmppReceiverServlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>xmppreceiver</servlet-name>
    <url-pattern>/_ah/xmpp/message/chat/</url-pattern>
  </servlet-mapping>
</web-app>

The URL path /_ah/xmpp/message/chat is reserved for XMPP messages to be sent to. Similarly, the paths /_ah/xmpp/presence/available/ and /_ah/xmpp/presence/unavailable are reserved for XMPP presence notifications.

Presto! We have a working (and amusing) XMPP bot! For the full source code, including the bits we left out for space reasons, see the next section.

Source

The Python source for the full implementation of the crowdguru sample application is available in our samples repository.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.