Google is committed to advancing racial equity for Black communities. See how.

Depth API developer guide for Android NDK

Learn how to use the Depth API in your own apps.

Depth API-supported devices

Only devices that are depth-supported should be able to discover depth-required apps in the Google Play Store. Discovery should be restricted to depth-supported devices when:

  • A core part of the experience relies on depth
  • There is no graceful fallback for the parts of the app that use depth

To restrict distribution of your app in the Google Play Store to devices that support the Depth API, add the following line to your AndroidManifest.xml:

<uses-feature android:name="com.google.ar.core.depth" />

Check if Depth API is supported

In a new ARCore session, check whether a user's device supports the Depth API.

CHECK(ArSession_create(env, context, &ar_session_) == AR_SUCCESS);

int32_t is_depth_supported = 0;
ArSession_isDepthModeSupported(ar_session_, AR_DEPTH_MODE_AUTOMATIC,
                               &is_depth_supported);
ArConfig* ar_config = nullptr;
ArConfig_create(ar_session_, &ar_config);
if (is_depth_supported) {
  ArConfig_setDepthMode(ar_session_, ar_config, AR_DEPTH_MODE_AUTOMATIC);
}
CHECK(ArSession_configure(ar_session_, ar_config) == AR_SUCCESS);
ArConfig_destroy(ar_config);

Retrieve depth maps

Call ArFrame_acquireDepthImage() to get the depth map for the current frame.

// Retrieve the depth map for the current frame, if available.
ArImage* depth_image = nullptr;
// If a depth map is available, use it here.
if (ArFrame_acquireDepthImage(&session, &frame, &depth_image) != AR_SUCCESS) {
  // No depth image received for this frame.
  // This normally means that depth data is not available yet.
  // Depth data will not be available if there are no tracked
  // feature points. This can happen when there is no motion, or when the
  // camera loses its ability to track objects in the surrounding
  // environment.
  return;
}

Visualize depth data and occlude virtual objects

The image provided through ArFrame_acquireDepthImage() in the preceding code sample provides the raw image buffer, which can be passed to a fragment shader for usage on the GPU for each rendered object to be occluded. Keep in mind that this image is oriented in AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES, and can be changed to AR_COORDINATES_2D_TEXTURE_NORMALIZED by calling ArFrame_transformCoordinates2d().

Once the depth map is accessible within an object shader, these depth measurements can be accessed directly for occlusion handling.

Parse the depth information for the current frame

As shown in the following code, the helper functions DepthGetMillimeters() and DepthGetVisibility() can be used in a fragment shader to access the depth information for the current screen position. This can then be used to selectively occlude parts of the rendered object.

// Use DepthGetMillimeters() and DepthGetVisibility() to parse the depth map
// for a given pixel, and compare against the depth of the object to render.
float DepthGetMillimeters(in sampler2D depth_texture, in vec2 depth_uv) {
  // Depth is packed into the red and green components of its texture.
  // The texture is a normalized format, storing millimeters.
  vec3 packedDepthAndVisibility = texture2D(depth_texture, depth_uv).xyz;
  return dot(packedDepthAndVisibility.xy, vec2(255.0, 256.0 * 255.0));
}

// Return a value representing how visible or occluded a pixel is relative
// to the depth map. The range is 0.0 (not visible) to 1.0 (completely
// visible).
float DepthGetVisibility(in sampler2D depth_texture, in vec2 depth_uv,
                         in float asset_depth_mm) {
  float depth_mm = DepthGetMillimeters(depth_texture, depth_uv);

  // Instead of a hard Z-buffer test, allow the asset to fade into the
  // background along a 2 * kDepthTolerancePerMm * asset_depth_mm
  // range centered on the background depth.
  const float kDepthTolerancePerMm = 0.015f;
  float visibility_occlusion = clamp(0.5 * (depth_mm - asset_depth_mm) /
    (kDepthTolerancePerMm * asset_depth_mm) + 0.5, 0.0, 1.0);

 // Use visibility_depth_near to set the minimum depth value. If using
 // this value for occlusion, avoid setting it too close to zero. A depth value
 // of zero signifies that there is no depth data to be found.
  float visibility_depth_near = 1.0 - InverseLerp(
      depth_mm, /*min_depth_mm=*/150.0, /*max_depth_mm=*/200.0);

  // Use visibility_depth_far to set the maximum depth value. If the depth
  // value is too high (outside the range specified by visibility_depth_far),
  // the virtual object may get inaccurately occluded at further distances
  // due to too much noise.
  float visibility_depth_far = InverseLerp(
      depth_mm, /*min_depth_mm=*/7500.0, /*max_depth_mm=*/8000.0);

  const float kOcclusionAlpha = 0.0f;
  float visibility =
      max(max(visibility_occlusion, kOcclusionAlpha),
          max(visibility_depth_near, visibility_depth_far));

  return visibility;
}

Simulate occlusion

Simulate occlusion in the body of the fragment shader.

Use the following code to update the object's alpha channel based on the depth, which simulates occlusion:

const float kMetersToMillimeters = 1000.0;

float asset_depth_mm = v_ViewPosition.z * kMetersToMillimeters * -1.;

// Computes the texture coordinates to sample from the depth map.
vec2 depth_uvs = (u_DepthUvTransform * vec3(v_ScreenSpacePosition.xy, 1)).xy;

gl_FragColor.a *= DepthGetVisibility(u_DepthTexture, depth_uvs, asset_depth_mm);

Conventional versus alternative implementations of occlusion rendering

The hello_ar_c sample app implements per-object, forward-pass rendering. As each object is rendered, the app uses the depth map to determine whether certain pixels in the virtual content are visible. If the pixels are not visible, they will be clipped, simulating occlusion on the user’s device.

An alternative way to render occlusion is to set up a two-pass rendering configuration. The first (render) pass renders all of the virtual content into an intermediary buffer. The second pass uses the depth map to combine the virtual content with the real world camera. The efficiency of each approach depends on the complexity of the scene and other app-specific considerations. The two-pass approach requires no additional shader work.

Alternative uses of the Depth API

The hello_ar_c app uses the Depth API to create depth maps and simulate occlusion. Other uses for the depth API include:

  • Collisions: virtual objects bouncing off walls after a user throws them
  • Distance measurement
  • Re-lighting a scene
  • Re-texturing existing objects: turning a floor into lava
  • Depth-of-field effects: blurring out the background or foreground
  • Environmental effects: fog, rain, and snow

For more detail and best practices for applying occlusion in shaders, check out the hello_ar_c sample app.

Understanding depth values

Given point A on the observed real-world geometry and a 2D point a representing the same point in the depth image, the value given by the Depth API at a is equal to the length of CA projected onto the principal axis. This can also be referred as the z-coordinate of A relative to the camera origin C. When working with the Depth API, it is important to understand that the depth values are not the length of the ray CA itself, but the projection of it.