# Integrate the Assistant into Your Project (Other Languages)

Follow the instructions in each section below to integrate the Google Assistant
into your project.

## gRPC bindings

The Google Assistant Service is built on top of [gRPC](http://www.grpc.io), a high
performance, open-source RPC framework. This framework is well-suited for
bidirectional audio streaming.

## Python

If you're using Python, get started using [this guide](https://developers.google.com/assistant/sdk/guides/service/python).

## C++

Take a look at our [C++ sample](https://github.com/googlesamples/assistant-sdk-cpp)
on GitHub.

## Node.js

Take a look at our [Node.js sample](https://github.com/googlesamples/assistant-sdk-nodejs)
on GitHub.

## Android Things

Interested in embedded devices? Check out the Assistant SDK
[sample](https://github.com/androidthings/sample-googleassistant) for
[Android Things](https://developer.android.com/things/).

## Other languages

- Clone the [googleapis](https://github.com/googleapis/googleapis) repository to get the protocol buffer interface definitions for the [Google Assistant Service API](https://github.com/googleapis/googleapis/blob/master/google/assistant/embedded/v1alpha2/embedded_assistant.proto).
- Follow the [gRPC documentation](http://www.grpc.io/docs/) to generate gRPC bindings for your language of choice
- Follow the steps in the sections below.

### Authorize and authenticate your Google account to work with the Assistant

The next step is to authorize your device to talk with the Google Assistant
using your Google account.

### Obtain OAuth tokens with the Assistant SDK scope

The Assistant SDK uses OAuth 2.0 access tokens to authorize your device to
connect with the Assistant.

When prototyping, you can use the authorization tool to easily generate OAuth2.0
credentials from the `client_secret_<client-id>.json` file generated when
[registering your device model](https://developers.google.com/assistant/sdk/guides/service/python/embed/register-device#download-credentials).

Do the following to generate the credentials:

1. Use a [Python virtual environment](https://docs.python.org/3/library/venv.html)
   to isolate the authorization tool and its dependencies from the system Python packages.

       sudo apt-get update
       sudo apt-get install python3-dev python3-venv # Use python3.4-venv if the package cannot be found.
       python3 -m venv env
       env/bin/python -m pip install --upgrade pip setuptools wheel
       source env/bin/activate

2. Install the authorization tool:

   ```
   python -m pip install --upgrade google-auth-oauthlib[tool]
   ```
3. Run the tool. Remove the `--headless` flag if you are running this from a terminal on the device (not an SSH session):

   ```
   google-oauthlib-tool --client-secrets /path/to/client_secret_client-id.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless
   ```

When you are ready to integrate the authorization as part of the provisioning
mechanism of your device, read our guides for [Using OAuth 2.0 to Access Google APIs](https://developers.google.com/identity/protocols/OAuth2)
to understand how to obtain, persist and use OAuth access tokens to allow your
device to talk with the Assistant API.

Use the following when working through these guides:

- OAuth scope: https://www.googleapis.com/auth/assistant-sdk-prototype
- Supported OAuth flows:

  - (Recommended) [Installed apps](https://developers.google.com/identity/protocols/OAuth2InstalledApp)
  - [Web server applications](https://developers.google.com/identity/protocols/OAuth2#webserver)

  > [!NOTE]
  > **Note:** The [limited input device flow](https://developers.google.com/identity/protocols/OAuth2#device) is not supported.

Check out the [best practices on privacy and security](https://developers.google.com/assistant/sdk/guides/service/python/best-practices/privacy-and-security)
for recommendations on how to secure your device.

#### Authenticate your gRPC connection with OAuth tokens

Finally, put all the pieces together by reading how to [use token-based
authentication with Google](http://www.grpc.io/docs/guides/auth.html) to
authenticate the gRPC connection to the Assistant API.

### Register your device

Register your device model and instance either [manually](https://developers.google.com/assistant/sdk/reference/device-registration/register-device-manual)
or with the [registration tool](https://developers.google.com/assistant/sdk/guides/service/python/embed/register-device) (available
in Python).

### Implement a basic conversation dialog with the Assistant

1. Implement a [bidirectional streaming gRPC client](http://www.grpc.io/docs/guides/concepts.html#bidirectional-streaming-rpc) for the Google Assistant Service API.
2. Wait for the user to trigger a new request (e.g., wait for a GPIO interrupt from a button press).
3. Send an [`AssistRequest`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistrequest)
   message with the `config` field set (see [`AssistConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistconfig)).
   Make sure the `config` field contains the following:

   - The `audio_in_config` field, which specifies how to process the `audio_in` data that will be provided in subsequent requests (see [`AudioInConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#audioinconfig)).
   - The `audio_out_config` field, which specifies the desired format for the server to use when it returns `audio_out` messages (see [`AudioOutConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#audiooutconfig)).
   - The `device_config` field, which identifies the registered device to the Assistant (see [`DeviceConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#deviceconfig)).
   - The `dialog_state_in` field, which contains the `language_code` associated with the request (see [`DialogStateIn`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#dialogstatein)).
4. Start recording.

5. Send multiple outgoing [`AssistRequest`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistrequest)
   messages with audio data from the spoken query in the `audio_in` field.

6. Handle incoming [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse)
   messages.

7. Extract conversation metadata from the [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse)
   message. For example, from `dialog_state_out`, get the `conversation_state`
   and `volume_percentage` (see [`DialogStateOut`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#dialogstateout)).

8. Stop recording when receiving a [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse)
   with an [`event_type`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#eventtype)
   of `END_OF_UTTERANCE`.

9. Play back audio from the Assistant answer with audio data coming from the [`audio_out`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#audioout)
   field.

10. Take the `conversation_state` you extracted earlier and copy it
    into the [`DialogStateIn`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#dialogstatein)
    message in the `AssistConfig` for the next [`AssistRequest`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistrequest).

With this, you should be ready to make your first requests to the Google
Assistant through your device.

### Extend a conversation dialog with Device Actions

Extend the basic conversation dialog above to trigger the unique hardware
capabilities of your particular device:

1. In the incoming [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse) messages, extract the `device_action` field (see [`DeviceAction`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#deviceaction)).
2. Parse the JSON payload of the `device_request_json` field. Refer to the [Device Traits](https://developers.google.com/assistant/sdk/reference/traits) page for the list of supported traits. Each trait schema page shows a sample EXECUTE request with the device command(s) and parameters that are returned in the JSON payload.

### Get the transcript of the user request

If you have a display attached to the device, you might want to use it to
show the user request. To get this transcript, parse the `speech_results` field
in the [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse)
messages. When the speech recognition completes, this list will contain one [item](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#speechrecognitionresult)
with a `stability` set to 1.0.

### Get the text and/or visual rendering of the Assistant's response

If you have a display attached to the device, you might want to use it to
show the Assistant's plain text response to the user's request. This text is located
in the [`DialogStateOut.supplemental_display_text`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#dialogstateout)
field.

The Assistant supports visual responses via HTML5 for certain queries (*What
is the weather in Mountain View?* or *What time is it?* ). To enable this, set
the `screen_out_config` field in [`AssistConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistconfig).
The [`ScreenOutConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#screenoutconfig)
message has field `screen_mode` which should be set to [`PLAYING`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#screenmode).

The [`AssistResponse`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistresponse)
messages will then have field `screen_out` set. You can extract the HTML5 data (if present) from the
[`data`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#screenout) field.

### Submitting queries via text input

If you have a text interface (for example, a keyboard) attached to the device,
set the `text_query` field in the `config` field (see [`AssistConfig`](https://developers.google.com/assistant/sdk/reference/rpc/google.assistant.embedded.v1alpha2#assistconfig)).
Do not set the `audio_in_config` field.

### Troubleshooting

See the [Troubleshooting](https://developers.google.com/assistant/sdk/guides/service/troubleshooting)
page if you run into issues.