Creating a Smart Home App

Smart Home apps are structured differently than a traditional app for the Assistant. This is because how users trigger actions and the actions' conversations are handled for you, and all you need to do is handle Smart Home intents on your service.

The general steps in creating a Smart Home app are:

  1. Setup an OAuth 2.0 server for account linking.
  2. Create an Actions on Google developer project.
  3. Create an action package, declaring support for Smart Home intents.
  4. Provide fulfillment of Smart Home intents.
  5. Test and submit your app for approval.

Set up an OAuth 2.0 Server

A user's third-party OAuth 2 access token is passed in the "Authorization" header when intents are sent to your fulfillment. No execution can occur before accounts are linked, because device information is sent to the Assistant with the action.devices.SYNC intent, which requires account linking.

OAuth integration is described in the account linking overview.

Your app is expected to support multiple Google users connecting to the same agent user account, for example when users give access to other users in their household. If your services can't support multiple user connections, it should provide errors at account linking time.

Make sure that you have a public OAuth 2.0 server for authenticating users and apps. It should conform to the OAuth 2.0 Authorization Code flow documentation. Once set up, validate your OAuth 2.0 server by following these instructions.

Create a developer project and action package

Create a project and agent with the Actions SDK.

In your action package, define support for the Smart Home intents by declaring an action that fulfills the action.devices intent.

{
  "actions": [{
    "name": "actions.devices",
    "deviceControl": {
    },
    "fulfillment": {
      "conversationName": "automation"
    }
  }],
  "conversations": {
    "automation" :
    {
      "name": "automation",
      "url": "https://example.com/google-assistant/endpoint"
    }
  }
}

While you are defining your action package, keep in mind that Smart Home apps are typically invoked differently than regular apps. When devices are registered in HomeGraph, they are controlled directly through voice commands instead of the user talking to your app.

For example, the command Turn on the light may find the light in HomeGraph that was registered by your app, and send your app the EXECUTE command without directly invoking your app.

When registering your app in the Actions Console, the invocation and display names are required to submit your project for approval, but they are not spoken by the end user.

The following intents are contained in the action.devices namespace and you must provide fulfillment for them:

  • action.devices.SYNC - Requests the list of devices that the user has connected and are available for use
  • action.devices.QUERY - Queries for the current states of devices
  • action.devices.EXECUTE - Requests a command to execute on Smart Home devices. The new state should be provided in response if available. One EXECUTE intent can target multiple devices, with multiple commands.

Build fulfillment

Your fulfillment must process the Smart Home intents and return responses back to the Assistant as defined by the conversation webhook. You can use any language as long as you adhere to the request and response formats. Your fulfillment should minimize latency between the Assistant and your cloud API.

The following sections describe the request and response formats for communication between the Assistant and your app.

Conversation webhook differences for Smart Home apps

Smart Home intents focus on device targets rather than user state, so the following changes are applied to the Conversation Webhook:

  1. device and conversation are not provided; the device field in Conversation Protocol refers to the surface client, not the IoT targets.
  2. The user field is not provided. Authorization token will be included in the Authorization header of the HTTP request.
  3. Smart Home Intents use a single object in inputs, containing the intent value, and a payload object with automation-specific objects.
  4. The response likewise includes a payload object tailored to the Smart Home intent.
  5. All requests include a requestId field with an arbitrary identifier for debugging between Google and partner apps.

action.devices.SYNC

This intent is triggered at user setup or when a user reconnects through the app (to reconnect or disconnect) when necessary to resync devices in batch (for example, when new traits are added).

To update users' devices without needing to unlink and relink their account, use Request Sync.

When this intent is triggered the Assistant sends a request to your fulfillment, which should respond with the following:

  • Required. Return all devices. Devices that are temporarily unavailable should be included, otherwise an offline device will no longer be recognized.
  • Required. Return the attributes of each device.
Request
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": "action.devices.SYNC"
  }]
}
Response
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "agentUserId": "1836.15267389",
    "devices": [{
      "id": "123",
      "type": "action.devices.types.OUTLET",
      "traits": [
        "action.devices.traits.OnOff"
      ],
      "name": {
        "defaultNames": ["My Outlet 1234"],
        "name": "Night light",
        "nicknames": ["wall plug"]
      },
      "willReportState": true,
      "deviceInfo": {
        "manufacturer": "lights-out-inc",
        "model": "hs1234",
        "hwVersion": "3.2",
        "swVersion": "11.4"
      },
      "customData": {
        "fooValue": 74,
        "barValue": true,
        "bazValue": "foo"
      }
    },{
      "id": "456",
      "type": "action.devices.types.LIGHT",
        "traits": [
          "action.devices.traits.OnOff", "action.devices.traits.Brightness",
          "action.devices.traits.ColorTemperature",
          "action.devices.traits.ColorSpectrum"
        ],
        "name": {
          "defaultNames": ["lights out inc. bulb A19 color hyperglow"],
          "name": "lamp1",
          "nicknames": ["reading lamp"]
        },
        "willReportState": true,
        "attributes": {
          "temperatureMinK": 2000,
          "temperatureMaxK": 6500
        },
        "deviceInfo": {
          "manufacturer": "lights out inc.",
          "model": "hg11",
          "hwVersion": "1.2",
          "swVersion": "5.4"
        },
        "customData": {
          "fooValue": 12,
          "barValue": false,
          "bazValue": "bar"
        }
      }]
  }
}

Request format

  • requestId: String. Required. Id of request for ease of tracing
  • inputs:
    • intent: String. Required. action.devices.SYNC
    • (No payload)

Response format

  • payload:
    • errorCode: String. Optional. For systematic errors on SYNC.
    • debugString: string. Optional. Detailed error which will never be presented to users but may be logged or used during development.
    • agentUserId: String (up to 256 bytes). Optional. Reflects the unique (and immutable) user ID on the agent's platform. The string is opaque to Google, so if there's an immutable form vs a mutable form on the agent side, use the immutable form (e.g. an account number rather than email).
    • devices: Array<Object>. Array of devices. Zero or more devices are returned (zero devices meaning the user has no devices, or has disconnected them all). Each device has the following properties:
      • id: String. Required. The ID of the device in the partner's cloud. This must be unique for the user and for the partner, as in cases of sharing we may use this to dedupe multiple views of the same device. It should be immutable for the device; if it changes, the Assistant will treat it as a new device.
      • type: String. Required. The hardware type of device (for example, action.devices.types.LIGHT). See the full list of device types.
      • traits: Array<String>. Required. List of traits this device supports (for example, action.devices.traits.OnOff). This defines the commands, attributes, and states that the device has. See the full list of device traits.
      • name: Object. Required. Names of this device. Note that while individual fields are optional, each device must have at least one name.
        • defaultNames: Array<String>. Optional. List of names provided by the partner rather than the user, often manufacturer names, SKUs, etc.
        • name: String. Optional. Primary name of the device, generally provided by the user. This is also the name the Assistant will prefer to describe the device in responses.
        • nicknames: Array<String>. Optional. Additional names provided by the user for the device.
      • willReportState: Boolean. Required. Indicates whether this device will have its states updated by the Real Time Feed. (TRUE to use the Real Time Feed for reporting state, and FALSE to use the polling model.)
      • roomHint: String. Optional. If the partner's cloud configuration includes placing devices in rooms, the name of the room can be provided here; the Assistant will attempt to align the room with those in the HomeGraph.
      • structureHint: String. Optional. As roomHint, for structures that users set up in the partner's system. Structures are houses, buildings, etc -- containers for rooms.
      • deviceInfo: Object. Optional. Contains fields describing the device for use in one-off logic if needed (e.g. 'broken firmware version X of light Y requires adjusting color', or 'security flaw requires notifying all users of firmware Z').
        • manufacturer: String. Especially useful when the partner is a hub for other devices. Google may provide a standard list of manufacturers here so that e.g. TP-Link and Smartthings both describe 'osram' the same way.
        • model: String. The model or SKU identifier of the particular device.
        • hwVersion: String. Specific version number attached to the hardware if available.
        • swVersion: String. Specific version number attached to the software/firmware, if available.
      • attributes: Object. Optional, aligned with per-trait attributes as in Attributes below. Right-hand values are string | int | boolean | number.
      • customData: Object. Optional; this is a special object defined by the partner which will be attached to future QUERY and EXECUTE requests. Partners can use this object to store additional information about the device to improve performance or routing within their cloud, such as the global region of the device. Data in this object has a few constraints:
        • No Personally Identifiable Information.
        • Data should change rarely, akin to other attributes -- so this should not contain real-time state.
        • The total object is limited to 512 bytes per device.

Request Sync

Request Sync will tell Google to make a SYNC call for any Google user with devices that have the specified agentUserId associated with them (which you sent in the original SYNC request). All users linked to this identifier will receive a SYNC request.

Do the following to trigger a SYNC request to your fulfillment. This allows you to update users' devices without unlinking and relinking their account.

  1. In the Cloud Platform Console, go to the Projects page. Select the project that matches your Smart Home project id.

  2. Enable the Google HomeGraph API.

  3. Generate an API key. From the left navbar, select Credentials under APIs & Services. Click the Create Credentials button and select API key.

  4. Copy the API key. Use this API key and the agentUserId in your request. Here's an example of how to make the request in the command line using curl, as a test:

    curl -i -s -k -X POST -H "Content-Type: application/json" -d "{agent_user_id: \"agentUserId\"}" \
      "https://homegraph.googleapis.com/v1/devices:requestSync?key=API_KEY"

action.devices.QUERY

This intent queries for the current states of devices from the partner. It is used for queries where truly real-time accuracy is required (for example, the status of a door lock). Only states are returned when action.devices.QUERY is triggered. To update properties or traits or other persistent elements of the device, action.devices.SYNC is used.

Request
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": "action.devices.QUERY",
    "payload": {
      "devices": [{
        "id": "123",
        "customData": {
          "fooValue": 74,
          "barValue": true,
          "bazValue": "foo"
        }
      },{
        "id": "456",
        "customData": {
          "fooValue": 12,
          "barValue": false,
          "bazValue": "bar"
        }
      }]
    }
  }]
}
Response
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "devices": {
      "123": {
        "on": true,
        "online": true
      },
      "456": {
        "on": true,
        "online": true,
        "brightness": 80,
        "color": {
          "name": "cerulean",
          "spectrumRGB": 31655
        }
      }
    }
  }
}

Request format

  • requestId: String. Required. Id of request for ease of tracing
  • inputs:
    • intent: String. Required. action.devices.QUERY
    • payload: Object. Required
      • devices: Array<Object>. Required.
        • id: Required. Partner ID to query, as per the id provided in SYNC.
        • customData: Optional. If the opaque customData object is provided in SYNC, it's sent here.

Response format

  • payload:
    • errorCode: String. Optional. An error code for the entire transaction -- for auth failures and partner system unavailability. For individual device errors use the errorCode within the device object.
    • debugString: string. Optional. Detailed error which will never be presented to users but may be logged or used during development.
    • devices: Object. Map of devices. Each property has the following name and value:
      • <id>: Object. Required. Maps partner device ID to object of state properties, as defined in States section below.
      • errorCode: String. Optional. Expanding ERROR state if needed from the preset error codes, which will map to the errors presented to users.
      • debugString: string. Optional. Detailed error which will never be presented to users but may be logged or used during development.

action.devices.EXECUTE

This intent is triggered to provide commands to execute on Smart Home devices. The new state should be provided in response if available. One triggered intent can target multiple devices, with multiple commands. For example, a triggered intent may set both brightness and color on a set of lights or may set multiple lights each to a different color.

Request
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": "action.devices.EXECUTE",
    "payload": {
      "commands": [{
        "devices": [{
          "id": "123",
          "customData": {
            "fooValue": 74,
            "barValue": true,
            "bazValue": "sheepdip"
          }
        },{
          "id": "456",
          "customData": {
            "fooValue": 36,
            "barValue": false,
            "bazValue": "moarsheep"
          }
        }],
        "execution": [{
          "command": "action.devices.commands.OnOff",
          "params": {
            "on": true
          }
        }]
      }]
    }
  }]
}
Response
{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "commands": [{
      "ids": ["123"],
      "status": "SUCCESS",
      "states": {
        "on": true,
        "online": true
      }
    },{
      "ids": ["456"],
      "status": "ERROR",
      "errorCode": "deviceTurnedOff"
    }]
  }
}

Request format

  • requestId: String. Required. Id of request for ease of tracing
  • inputs
    • intent: String. Required. action.devices.EXECUTE
    • payload:
      • commands: Array<Object>. Required. Each object contains one or more devices to target with the attached commands.
        • devices: Array<Object>code>. Required.
          • id: Required. Partner ID to query, as per the id provided in SYNC.
          • customData: Optional. If the opaque customData object is provided in SYNC, it's sent here.
        • execution: Array<Object>. Required. The ordered list of execution commands for the attached ids.
          • command: String. Required. The command (see below) to execute, with (usually) accompanying parameters.
          • params: Map<string, Object> Optional, but aligned with the parameters for each command (below).
            • <name>: String. Required. The name of the param.
            • <value>: String | Number | Boolean

Response format

  • payload:
    • errorCode: String. Optional. An error code for the entire transaction -- for auth failures and partner system unavailability. For individual device errors use the errorCode within the device object.
    • debugString: string. Optional. Detailed error which will never be presented to users but may be logged or used during development.
    • commands: Array<Object>. Required. Each object contains one or more devices with response details. N.B. These may not be grouped the same way as in the request. For example, the request might turn 7 lights on, with 3 lights succeeding and 4 failing, thus with two groups in the response.
      • ids: Array<String>. Required. Partner device IDs of the response
      • status: String. Required. Current status types:
        • SUCCESS - confirmed that the command(s) has/have succeeded.
        • PENDING - commands are enqueued but expected to succeed.
        • OFFLINE - target devices(s) in offline state or unreachable.
        • ERROR - unable to perform the commands.
      • errorCode: String. Optional. Expanding ERROR state if needed from the preset error codes, which will map to the errors presented to users.
      • debugString: string. Optional. Detailed error which will never be presented to users but may be logged or used during development.
      • states: Object. Optional, but aligned with per-trait states as in Attributes below. These are the states _after_ execution, if available.
        • <name>: String. Required. The name of the param.
        • <value>: String | Number | Boolean

Error responses

These are the error codes that can you can returned in your responses. This list is likely to be expanded, and of course not all errors are relevant in all contexts. These error codes generally map to the assistant speech response we give for devices.

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "errorCode": "notSupported"
  }
}
  • authExpired: Credentials have expired.
  • authFailure: General failure to authenticate.
  • deviceOffline: The target is unreachable.
  • timeout: Internal timeout.
  • deviceTurnedOff: The device is known to be turned hard off (if distinguishable from unreachable).
  • deviceNotFound: The device doesn't exist on the partner's side. This normally indicates a failure in data synchronization or a race condition.
  • valueOutOfRange: The range in parameters is out of bounds.
  • notSupported: The command or its parameters are unsupported (this should generally not happen, as traits and business logic should prevent it).
  • protocolError: Failure in processing the request.
  • unknownError: Everything else, although anything that throws this should be replaced with a real error code.