Lighting Estimation developer guide for Android

Learn how to use Lighting Estimation in your own apps.

Configure the API once per session with the appropriate mode

Configure LightingEstimation once per session for the mode you want to use.

// Configure the session with the Lighting Estimation API in ENVIRONMENTAL_HDR mode.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.ENVIRONMENTAL_HDR);
session.configure(config);

// Configure the session with the Lighting Estimation API in AMBIENT_INTENSITY mode.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.AMBIENT_INTENSITY);
session.configure(config);

// Configure the session with the Lighting Estimation API turned off.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.DISABLED);
session.configure(config);

Configure ENVIRONMENTAL_HDR mode

To configure ENVIRONMENTAL_HDR mode, get the light estimate for each frame, then get the environmental HDR lighting components you want to use.

update() {

  // Get the current frame.
  Frame frame = session.update();

  // Get the light estimate for the current frame.
  LightEstimate lightEstimate = frame.getLightEstimate();

  // Get intensity and direction of the main directional light from the current light estimate.
  float[] intensity = lightEstimate.getEnvironmentalHdrMainLightIntensity(); // note - currently only out param.
  float[] direction = lightEstimate.getEnvironmentalHdrMainLightDirection();
  app.setDirectionalLightValues(intensity, direction); // app-specific code.

  // Get ambient lighting as spherical harmonics coefficients.
  float[] harmonics = lightEstimate.getEnvironmentalHdrAmbientSphericalHarmonics();
  app.setAmbientSphericalHarmonicsLightValues(harmonics); // app-specific code.

  // Get HDR environmental lighting as a cubemap in linear color space.
  Image[] lightmaps = lightEstimate.getEnvironmentalHdrCubeMap();
  for (int i = 0; i < lightmaps.length /*should be 6*/; ++i) {
    app.UploadToTexture(i, lightmaps[i]);  // app-specific code.
  }
}

Configure AMBIENT_INTENSITY mode

If you're planning to use the color correction component of AMBIENT_INTENSITY mode, first avoid allocation of color correction on every frame.

// Avoid allocation on every frame.
float[] colorCorrection = new float[4];

Get the light estimate for each frame, and then get ambient intensity components you want to use.

update() {

  // Get the current frame.
  Frame frame = session.update();

  // Get the light estimate for the current frame.
  LightEstimate lightEstimate = frame.getLightEstimate();

  // Get the pixel intensity of AMBIENT_INTENSITY mode.
  float[] pixelIntensity = lightEstimate.getPixelIntensity();

  // Get the pixel color correction of AMBIENT_INTENSITY mode.
  lightEstimate.getColorCorrection(colorCorrection, 0);

}

Ensuring energy conservation with Environmental HDR APIs

Energy conservation is the principle that light reflected from a surface will never be more intense than it was before it hit the surface. This rule is enforced in physically-based rendering, but is usually omitted from legacy rendering pipelines used in video games and mobile apps.

If you're using a physically-based rendering pipeline with Environmental HDR light estimation, simply ensure physically-based materials are used in your virtual objects.

If you aren't using a physically-based pipeline, however, you have a couple of options:

  • The most ideal solution for this is to migrate to a physically-based pipeline.

  • If that isn't possible, however, a good workaround is to multiply the albedo value from a non-physically-based material by an energy conservation factor. This can make sure at least the BRDF shading model can be converted into physically-based. Each BRDF has a different factor -- for example, for a diffuse reflection it is 1/Pi.

Send feedback about...