Platform-specific guides
Android (Kotlin/Java)
Android NDK (C)
iOS
Unity (AR Foundation)
The Scene Semantics API enables developers to understand the scene surrounding the user, which is needed for many high-quality AR experiences. Built on an ML model, the Scene Semantics API provides real-time semantic information, which complements existing geometric information in ARCore.
Given an image of an outdoor scene, the API returns a label for each pixel across a set of useful semantic classes, such a sky, building, tree, road, sidewalk, vehicle, person, and more. In addition to pixel labels, the Scene Semantics API also offers confidence values for each pixel label and an easy-to-use way to query the prevalence of a given label in an outdoor scene.
From left to right, examples of an input image, the semantic image of pixel labels, and the corresponding confidence image:
With the Scene Semantics API, developers can identify specific scene components, such as roads and sidewalks to help guide a user through an unfamiliar city, people and vehicles to render occlusions on dynamic objects, sky to create a sunset at any time of the day, and buildings to modify their appearance and anchor virtual objects.
Semantic labels and quality
The Scene Semantics API provides multiple labels, each with a corresponding quality or reliability. Generally, the ML model is better able to predict classes of larger, more common objects/surfaces than classes of smaller or more rare objects/surfaces. The classes can be grouped into the following quality tiers, ranked from higher to lower:
| Semantic label quality tiers | |
|---|---|
| Main scene components |
|
| Major scene details |
|
| Minor scene details |
|
Device compatibility
The Scene Semantics API shares the same list of supported devices as the Depth API. Please refer to the ARCore supported devices page for an up-to-date-list of devices that support both APIs.
Supported use cases
The Scene Semantics API is designed for use in the following scenarios:
Outdoor scenes: Supports outdoor scenes only and is not intended for indoor use cases.
Portrait orientation: Should only be used in the device's default orientation mode (i.e. portrait). The quality of semantic labels is not guaranteed for landscape mode.