Lighting Estimation developer guide for Android NDK

Learn how to use Lighting Estimation in your own apps.

Configure the API once per session with the appropriate mode

Configure LightingEstimation once per session for the mode you want to use.

// Configure the session's lighting estimation mode for AR_LIGHT_ESTIMATION_MODE_ENVIRONMENTAL_HDR.
ArConfig* config = nullptr;
ArConfig_create(ar_session_, &config);
ArSession_getConfig(ar_session_, config);
ArConfig_setLightEstimationMode(ar_session_, config,
                                AR_LIGHT_ESTIMATION_MODE_ENVIRONMENTAL_HDR);
ArSession_configure(ar_session_, config);
ArConfig_destroy(config);

// Configure the session's lighting estimation mode for AR_LIGHT_ESTIMATION_MODE_AMBIENT_INTENSITY.
ArConfig* config = nullptr;
ArConfig_create(ar_session_, &config);
ArSession_getConfig(ar_session_, config);
ArConfig_setLightEstimationMode(ar_session_, config,
                                AR_LIGHT_ESTIMATION_MODE_AMBIENT_INTENSITY);
ArSession_configure(ar_session_, config);
ArConfig_destroy(config);

// Disable the session's lighting estimation mode.
ArConfig* config = nullptr;
ArConfig_create(ar_session_, &config);
ArSession_getConfig(ar_session_, config);
ArConfig_setLightEstimationMode(ar_session_, config,
                                AR_LIGHT_ESTIMATION_MODE_DISABLED);
ArSession_configure(ar_session_, config);
ArConfig_destroy(config);

Configure AR_LIGHT_ESTIMATION_MODE_ENVIRONMENTAL_HDR mode

To configure AR_LIGHT_ESTIMATION_MODE_ENVIRONMENTAL_HDR mode, get the light estimate for each frame, then get the environmental HDR lighting components you want to use.

void update() {

  // Get the current frame.
  if (ArSession_update(ar_session_, ar_frame_) != AR_SUCCESS) {
    LOGE("ArSession_update error");
    return;
  }

  // Get the light estimate for the current frame.
  ArLightEstimate* ar_light_estimate;
  ArLightEstimateState ar_light_estimate_state;
  ArLightEstimate_create(ar_session_, &ar_light_estimate);

  ArFrame_getLightEstimate(ar_session_, ar_frame_, ar_light_estimate);
  ArLightEstimate_getState(ar_session_, ar_light_estimate,
                           &ar_light_estimate_state);

   // Check that the light estimate is valid before proceeding.
   if (ar_light_estimate_state != AR_LIGHT_ESTIMATE_STATE_VALID) {
     LOGE("ArSession_update error");
     return;
  }

  // Get intensity and direction of the main directional light from the current light estimate.
  float direction[3];
  ArLightEstimate_getEnvironmentalHdrMainLightDirection(
  ar_session_, ar_light_estimate, direction);

  float intensity[3];
  ArLightEstimate_getEnvironmentalHdrMainLightIntensity(
  ar_session_, ar_light_estimate, intensity);

  // Get ambient lighting as spherical harmonics coefficients.
  float ambient_spherical_harmonics[27];
  if (ar_light_estimate_state == AR_LIGHT_ESTIMATE_STATE_VALID) {
    ArLightEstimate_getEnvironmentalHdrAmbientSphericalHarmonics(
        ar_session_, ar_light_estimate, ambient_spherical_harmonics);
  }

  // Get HDR environmental lighting as a cubemap in linear color space.
  if (ar_light_estimate_state == AR_LIGHT_ESTIMATE_STATE_VALID) {
    ArImageCubemap cubemap_textures;
    ArLightEstimate_acquireEnvironmentalHdrCubemap(ar_session_, ar_light_estimate,
                                                   cubemap_textures);
    int width = -1;
    int height = -1;
    int32_t format = -1;
    for (int i = 0; i < 6; ++i) {
      ArImage* image_ptr = cubemap_textures[i];
      // We can access the cubemap texture data through ArImage APIs.
      ArImage_getWidth(session, image_ptr, &width);
      ArImage_getHeight(session, image_ptr, &height);
      ArImage_getImageFormat(session, image_ptr, &format);
      // Acquired image must be released with ArImage_release once it is no longer
      // needed.
      ArImage_release(image_ptr);
    }
}

Configure AR_LIGHT_ESTIMATION_MODE_AMBIENT_INTENSITY mode

If you're planning to use the color correction component of AMBIENT_INTENSITY mode, first avoid allocation of color correction on every frame.

// Avoid allocation on every frame.
float color_correction[4];

Get the light estimate for each frame, and then get ambient intensity components you want to use.

void update() {

  // Get the current frame.
  if (ArSession_update(ar_session_, ar_frame_) != AR_SUCCESS) {
    LOGE("ArSession_update error");
  }

  // Get the light estimate for the current frame.
  ArLightEstimate* ar_light_estimate;
  ArLightEstimateState ar_light_estimate_state;
  ArLightEstimate_create(ar_session_, &ar_light_estimate);

  ArFrame_getLightEstimate(ar_session_, ar_frame_, ar_light_estimate);
  ArLightEstimate_getState(ar_session_, ar_light_estimate,
                           &ar_light_estimate_state);

  // Get the pixel intensity of AR_LIGHT_ESTIMATION_MODE_AMBIENT_INTENSITY mode.
  float pixel_intensity;
  ArLightEstimate_getPixelIntensity (ar_session_, ar_light_estimate,
                                     &pixel_intensity);

  // Get the pixel color correction of AR_LIGHT_ESTIMATION_MODE_AMBIENT_INTENSITY mode.
  float color_correction[4];
  ArLightEstimate_getColorCorrection (ar_session_, ar_light_estimate,
                                      color_correction);

}

Ensuring energy conservation with Environmental HDR APIs

Energy conservation is the principle that light reflected from a surface will never be more intense than it was before it hit the surface. This rule is enforced in physically-based rendering, but is usually omitted from legacy rendering pipelines used in video games and mobile apps.

If you're using a physically-based rendering pipeline with Environmental HDR light estimation, simply ensure physically-based materials are used in your virtual objects.

If you aren't using a physically-based pipeline, however, you have a couple of options:

  • The most ideal solution for this is to migrate to a physically-based pipeline.

  • If that isn't possible, however, a good workaround is to multiply the albedo value from a non-physically-based material by an energy conservation factor. This can make sure at least the BRDF shading model can be converted into physically-based. Each BRDF has a different factor -- for example, for a diffuse reflection it is 1/Pi.

Send feedback about...