Understand the user's environment with the Scene Semantics API

Platform-specific guides

Scene semantics

The Scene Semantics API enables developers to understand the scene surrounding the user, which is needed for many high-quality AR experiences. Built on an ML model, the Scene Semantics API provides real-time semantic information, which complements existing geometric information in ARCore.

Given an image of an outdoor scene, the API returns a label for each pixel across a set of useful semantic classes, such a sky, building, tree, road, sidewalk, vehicle, person, and more. In addition to pixel labels, the Scene Semantics API also offers confidence values for each pixel label and an easy-to-use way to query the prevalence of a given label in an outdoor scene.

From left to right, examples of an input image, the semantic image of pixel labels, and the corresponding confidence image:

Example of semantic images

With the Scene Semantics API, developers can identify specific scene components, such as roads and sidewalks to help guide a user through an unfamiliar city, people and vehicles to render occlusions on dynamic objects, sky to create a sunset at any time of the day, and buildings to modify their appearance and anchor virtual objects.

Semantic labels and quality

The Scene Semantics API provides multiple labels, each with a corresponding quality or reliability. Generally, the ML model is better able to predict classes of larger, more common objects/surfaces than classes of smaller or more rare objects/surfaces. The classes can be grouped into the following quality tiers, ranked from higher to lower:

Semantic label quality tiers
Main scene components
  • sky
  • building
  • tree
  • road
  • vehicle
Major scene details
  • sidewalk
  • terrain
  • structure
  • water
Minor scene details
  • object
  • person

Device compatibility

The Scene Semantics API shares the same list of supported devices as the Depth API. Please refer to the ARCore supported devices page for an up-to-date-list of devices that support both APIs.

Supported use cases

The Scene Semantics API is designed for use in the following scenarios:

  1. Outdoor scenes: Supports outdoor scenes only and is not intended for indoor use cases.

  2. Portrait orientation: Should only be used in the device's default orientation mode (i.e. portrait). The quality of semantic labels is not guaranteed for landscape mode.