Catch up on all the news, sessions, and announcements from Google I/O 2021. Watch now.

Build your web app

A web app is the user interface (UI) for an Action that uses Interactive Canvas. You can use existing web technologies (such HTML, CSS, JavaScript, and WebAssembly) to design and develop your web app. For the most part, Interactive Canvas can render web content like a browser, but there are a few restrictions enforced for user privacy and security. Before you begin designing your UI, consider the design principles outlined in Design guidelines. We recommend using Firebase hosting to deploy your web app.

The HTML and JavaScript for your web app do the following:

This page goes over the recommended ways to build your web app, how to enable communication between your Conversational Action and your web app, and general guidelines and restrictions.

Although you can use any method to build your UI, Google recommends using the following libraries:

Architecture

Google strongly recommends using a single-page application architecture. This approach allows for optimal performance and supports continuous conversational user experience. Interactive Canvas can be used in conjunction with front-end frameworks like Vue, Angular, and React, which help with state management.

HTML file

The HTML file defines how your UI looks. This file also loads the Interactive Canvas API, which enables communication between your web app and your Conversational Action.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <title>Interactive Canvas Sample</title>

    <!-- Disable favicon requests -->
    <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;,">

    <!-- Load Interactive Canvas JavaScript -->
    <script src="https://www.gstatic.com/assistant/interactivecanvas/api/interactive_canvas.min.js"></script>

    <!-- Load PixiJS for graphics rendering -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/pixi.js/4.8.7/pixi.min.js"></script>

    <!-- Load Stats.js for fps monitoring -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/stats.js/r16/Stats.min.js"></script>

    <!-- Load custom CSS -->
    <link rel="stylesheet" href="css/main.css">
  </head>
  <body>
    <div id="view" class="view">
      <div class="debug">
        <div class="stats"></div>
        <div class="logs"></div>
      </div>
    </div>
    <!-- Load custom JavaScript after elements are on page -->
    <script src="js/log.js"></script>
    <script type="module" src="js/main.js"></script>
  </body>
</html>

Communicate between Conversational Action and web app

After you've built your web app and Conversational Action and loaded in the Interactive Canvas library in your web app file, you need to define how your web app and Conversational Action interact. To do this, modify the files that contain your web app logic.

action.js

This file contains the code to define callbacks and invoke methods through interactiveCanvas. Callbacks allow your web app to respond to information or requests from the Conversational Action, while methods provide a way to send information or requests to the Conversational Action.

Add interactiveCanvas.ready(callbacks); to your HTML file to initialize and register callbacks:

// action.js

/**
* This class is used as a wrapper for Google Assistant Canvas Action class
* along with its callbacks.
*/
export class Action {
 /**
  * @param  {Phaser.Scene} scene which serves as a container of all visual
  * and audio elements.
  */
 constructor(scene) {
   this.canvas = window.interactiveCanvas;
   this.gameScene = scene;
   const that = this;
   this.intents = {
     GUESS: function(params) {
       that.gameScene.guess(params);
     },
     DEFAULT: function() {
       // do nothing, when no command is found
     },
   };
 }

 /**
  * Register all callbacks used by the Interactive Canvas Action
  * executed during game creation time.
  */
 setCallbacks() {
   const that = this;
   // Declare the Interactive Canvas action callbacks.
   const callbacks = {
     onUpdate(data) {
       const intent = data[0].google.intent;
       that.intents[intent ? intent.name.toUpperCase() : 'DEFAULT'](intent.params);
     },
   };
   // Called by the Interactive Canvas web app once web app has loaded to
   // register callbacks.
   this.canvas.ready(callbacks);
 }
}

main.js

The main.js JavaScript module imports the files action.js and scene.js and creates instances of each of them when the web app loads. This module also registers callbacks for Interactive Canvas.

// main.js
import {Action} from './action.js';
import {Scene} from './scene.js';
window.addEventListener('load', () => {
  window.scene = new Scene();
  // Set Google Assistant Canvas Action at scene level
  window.scene.action = new Action(scene);
  // Call setCallbacks to register Interactive Canvas
  window.scene.action.setCallbacks();
});

scene.js

The scene.js file constructs the scene for your web app. The following is an excerpt from scene.js:

// scene.js

const view = document.getElementById('view');

// initialize rendering and set correct sizing
this.radio = window.devicePixelRatio;
this.renderer = PIXI.autoDetectRenderer({
  transparent: true,
  antialias: true,
  resolution: this.radio,
  width: view.clientWidth,
  height: view.clientHeight,
});
this.element = this.renderer.view;
this.element.style.width = `${this.renderer.width / this.radio}px`;
this.element.style.height = `${(this.renderer.height / this.radio)}px`;
view.appendChild(this.element);

// center stage and normalize scaling for all resolutions
this.stage = new PIXI.Container();
this.stage.position.set(view.clientWidth / 2, view.clientHeight / 2);
this.stage.scale.set(Math.max(this.renderer.width,
    this.renderer.height) / 1024);

// load a sprite from a svg file
this.sprite = PIXI.Sprite.from('triangle.svg');
this.sprite.anchor.set(0.5);
this.sprite.tint = 0x00FF00; // green
this.sprite.spin = true;
this.stage.addChild(this.sprite);

// toggle spin on touch events of the triangle
this.sprite.interactive = true;
this.sprite.buttonMode = true;
this.sprite.on('pointerdown', () => {
  this.sprite.spin = !this.sprite.spin;
});

Support touch interactions

Your Interactive Canvas Action can respond to your user's touch as well as their vocal inputs. Per the Interactive Canvas design guidelines, you should develop your Action to be "voice-first". That being said, some smart displays support touch interactions.

Supporting touch is similar to supporting conversational responses; however, instead of a vocal response from the user, your client-side JavaScript looks for touch interactions and uses those to change elements in the web app.

You can see an example of this in the sample, which uses the Pixi.js library:

// scene.js
…
this.sprite = PIXI.Sprite.from('triangle.svg');
…
this.sprite.interactive = true; // Enables interaction events
this.sprite.buttonMode = true; // Changes `cursor` property to `pointer` for PointerEvent
this.sprite.on('pointerdown', () => {
  this.sprite.spin = !this.sprite.spin;
});

Add more features

Now that you've learned the basics, you can enhance and customize your Action with Canvas-specific methods. This section explains how to implement these methods in your Interactive Canvas Action.

sendTextQuery()

The sendTextQuery() method sends text queries to the Conversational Action to programmatically match an intent. This sample uses sendTextQuery() to restart the triangle-spinning game when the user clicks a button. When the user clicks the "Restart game" button, sendTextQuery() sends a text query that matches the Restart game intent and returns a promise. This promise results in SUCCESS if the intent is triggered and BLOCKED if it is not. The following snippet matches the intent and handles the success and failure cases of the promise:

// scene.js
…
/**
* Handle game restarts
*/
async handleRestartGame() {
    console.log(`Request in flight`);
    this.button.texture = this.button.textureButtonDisabled;
    this.sprite.spin = false;
    const res = await this.action.canvas.sendTextQuery('Restart game');
    if (res.toUpperCase() !== 'SUCCESS') {
        console.log(`Request in flight: ${res}`);
        return;
    }
    console.log(`Request in flight: ${res}`);
    this.button.texture = this.button.textureButtonDisabled;
    this.sprite.spin = false;
}
...

If the promise results in SUCCESS, the Restart game webhook handler sends a Canvas response to your web app:

// index.js
…
app.handle('restart', conv => {
  conv.add(new Canvas({
    data: {
      command: 'RESTART_GAME'
    }
  }));
});
...

This Canvas response triggers the onUpdate() callback, which executes the code in the RESTART_GAME code snippet below:

// action.js
…
RESTART_GAME: (data) => {
    this.scene.button.texture = this.scene.button.textureButton;
    this.scene.sprite.spin = true;
    this.scene.sprite.tint = 0x00FF00; // green
    this.scene.sprite.rotation = 0;
},
...

OnTtsMark()

The OnTtsMark() callback is called when you include a <mark> tag with a unique name in your SSML response to the user. In the following snippets,OnTtsMark() synchronizes the web app's animation with the corresponding TTS output. When the Action has said to the user, "Sorry, you lost," the web app spells out the correct word and displays the letters to the user.

In the following example, the webhook handler revealWord includes a custom mark in the response to the user when they've lost the game:

// index.js
...
app.handle('revealWord', conv => {
  conv.add(new Simple(`<speak>Sorry, you lost.<mark name="REVEAL_WORD"/> The word is ${conv.session.params.word}.</speak>`));
  conv.add(new Canvas());
});
...

The following code snippet then registers the OnTtsMark() callback, checks the name of the mark, and executes the revealCorrectWord() function, which updates the web app:

// action.js
...
setCallbacks() {
    // declare Assistant Canvas Action callbacks
    const callbacks = {
        onTtsMark(markName) {
            if (markName === 'REVEAL_WORD') {
                // display the correct word to the user
                that.revealCorrectWord();
            }
        },
    }
    callbacks.onUpdate.bind(this);
}
...

setCanvasState()

The setCanvasState() method allows you to send state data from your Interactive Canvas web app to your fulfillment, and notifies Assistant that the web app has updated its state. The web app sends its updated state as a JSON object.

Calling setCanvasState() does not invoke an intent. After invoking setCanvasState(), if sendTextQuery() is invoked or the user query matches an intent in the conversation, the data that was set with setCanvasState() in the previous conversational turn is then available in subsequent turns of conversation. You can access this data through conv.context.canvas in your fulfillment.

In the following snippet, the web app uses setCanvasState() to send state data to fulfillment:

this.action.canvas.setCanvasState({ score: 150 })

The fulfillment code can then access the state, as shown in the following snippet:

app.handle('intent-name', conv => {
    console.log(conv.context.canvas.state)
})

Enable full-screen mode

By default, Interactive Canvas web apps include a header at the top of the screen with the name of your Action. You can use enableFullScreen to remove the header and replace it with a temporary toast that appears in the loading screen, which allows your user to have a full-screen experience while interacting with your Action. The toast message shows the Action's display name, the developer's name, and instructions for exiting the Action, and the toast color changes depending on what the user selects as a theme on their device.

Figure 1. A toast message on the loading screen for an Action.

If a user interacts with your Action frequently, the toast message temporarily stops appearing on the loading screen. If the user does not engage with your Action for a while, the toast message reappears when they launch the Action.

You can enable full-screen mode in your webhook or in a static prompt in Actions Builder.

To enable full-screen mode in your webhook, follow this step:

  1. Set the enableFullScreen field to true in the first canvas response returned by the webhook in a session. The following snippet is an example implementation using the Node.js client library:

     const { conversation, Canvas } = require('@assistant/conversation');
 const functions = require('firebase-functions');

 const app = conversation();

 app.handle('invocation_fullscreen', conv => {
   conv.add(new Canvas(
     {
       url: 'https://example-url.com',
       enableFullScreen: true
     }));
 });

 exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);

To enable full-screen mode in a static prompt in Actions Builder, follow these steps:

  1. Open your project in the Actions console.
  2. Click Develop in the navigation bar and open the prompt that includes the first canvas response.

  3. Set enable_full_screen to true, as shown in the following snippet:

     {
  "candidates": [
    {
      "canvas": {
        "url": "https://example-url.com",
        "enable_full_screen": true
      }
    }
  ]
}

Troubleshooting

While you can use the simulator in the Actions console to test your Interactive Canvas Action during development, you can also see errors that occur within your Interactive Canvas web app on users' devices in production. You can view these errors in your Google Cloud Platform logs.

To see these error messages in your Google Cloud Platform logs, follow these steps:

  1. Open your Actions project in the Actions console.
  2. Click Test in the top navigation.
  3. Click the View logs in Google Cloud Platform link.

Errors from your users' devices appear in chronological order in the logs viewer.

Error types

There are three types of web app errors you can see in the Google Cloud Platform logs:

  • Timeouts that occur when ready is not called within 10 seconds
  • Timeouts that occur when the promise returned by onUpdate() is not fulfilled within 10 seconds
  • JavaScript runtime errors that are not caught within your web app

View JavaScript error details

The details of JavaScript runtime errors within your web app aren't available by default. To see the details of JavaScript runtime errors, follow these steps:

  1. Ensure that you've configured the appropriate cross-origin resource sharing (CORS) HTTP response headers in your web app files. For more information, see Cross-origin resource sharing.
  2. Add crossorigin="anonymous" to your imported <script> tags in your HTML file, as shown in the following code snippet:
<script crossorigin="anonymous" src="<SRC>"></script>

Guidelines and restrictions

Take the following guidelines and restrictions into consideration as you develop your web app:

  • No cookies
  • No local storage
  • No geolocation
  • No camera usage
  • No audio or video recording
  • No popups
  • Stay under the 200 MB memory limit
  • Take the Action name header into account when rendering content (occupies upper portion of screen)
  • No styles can be applied to videos
  • Only one media element may be used at a time
  • No Web SQL database
  • No support for the SpeechRecognition interface of the Web Speech API.
  • Dark mode setting not applicable
  • Video playback is supported on smart displays. For more information on the supported media container formats and codecs, see Google Nest Hub codecs.

Cross-origin resource sharing

Because Interactive Canvas web apps are hosted in an iframe and the origin is set to null, you must enable cross-origin resource sharing (CORS) for your web servers and storage resources. This process allows your assets to accept requests from null origins.