Stabilize camera images on Android NDK (C)

ARCore now supports Electronic Image Stabilization (EIS), which helps produce a smooth camera preview. EIS achieves stabilization by observing phone movement using gyro and applying compensation homography mesh within the boundaries of camera texture that counters the minor shakes. EIS is only supported in the device's portrait orientation. All orientations will be supported in the 1.39.0 release of ARCore.

Query for EIS support and enable EIS

To enable EIS, configure your session to use AR_IMAGE_STABILIZATION_MODE_EIS. If the device doesn't support the EIS feature, this will cause an exception to be thrown from ARCore.

int enableEis = 0;
ArSession_isImageStabilizationModeSupported(
    ar_session, AR_IMAGE_STABILIZATION_MODE_EIS, &enableEis);
if (!enableEis) {
  return;
}
// Create a session config.
ArConfig* ar_config = NULL;
ArConfig_create(ar_session, &ar_config);

// Enable Electronic Image Stabilization.
ArConfig_setImageStabilizationMode(ar_session, ar_config, AR_IMAGE_STABILIZATION_MODE_EIS);
CHECK(ArSession_configure(ar_session, ar_config) == AR_SUCCESS);

// Release config resources.
ArConfig_destroy(ar_config);

Transform coordinates

When EIS is on, the renderer needs to use the modified device coordinates and matching texture coordinates that incorporate the EIS compensation when rendering the camera background. To get the EIS compensated coordinates, use ArFrame_transformCoordinates3d, using AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES as input and AR_COORDINATES_3D_EIS_NORMALIZED_DEVICE_COORDINATES as output to get 3D device coordinates and AR_COORDINATES_3D_EIS_TEXTURE_NORMALIZED as output to get 3D texture coordinates. For now, the only supported input coordinate type for ArFrame_transformCoordinates3d is AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES.

int kNumVertices = 4;
// Positions of the quad vertices in clip space (X, Y).
const GLfloat kVertices[] = {
    -1.0f, -1.0f, +1.0f, -1.0f, -1.0f, +1.0f, +1.0f, +1.0f,
};
float transformed_vertices_[4 * 3];
float transformed_uvs_[4 * 3];

ArFrame_transformCoordinates3d(
    session, frame, AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES,
    kNumVertices, kVertices,
    AR_COORDINATES_3D_EIS_NORMALIZED_DEVICE_COORDINATES,
    transformed_vertices_);
ArFrame_transformCoordinates3d(
    session, frame, AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES,
    kNumVertices, kVertices, AR_COORDINATES_3D_EIS_TEXTURE_NORMALIZED,
    transformed_uvs_);

glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_EXTERNAL_OES, camera_texture_id_);
glUseProgram(camera_program_);
glUniform1i(camera_texture_uniform_, 0);

// Set the vertex positions and texture coordinates.
glVertexAttribPointer(camera_position_attrib_, 3, GL_FLOAT, false, 0,
                      transformed_vertices_);
glVertexAttribPointer(camera_tex_coord_attrib_, 3, GL_FLOAT, false, 0,
                      transformed_uvs_);
glEnableVertexAttribArray(camera_position_attrib_);
glEnableVertexAttribArray(camera_tex_coord_attrib_);

When EIS is off, the output 3D coordinates are equivalent to their 2D counterparts, with z values set to produce no change.

Modify shaders

The 3D coordinates calculated about should be passed to background rendering shaders. The vertex buffers are now 3D with EIS:

layout(location = 0) in vec4 a_Position;
layout(location = 1) in vec3 a_CameraTexCoord;
out vec3 v_CameraTexCoord;
void main() {
  gl_Position = a_Position;
  v_CameraTexCoord = a_CameraTexCoord;
}

Additionally, the fragment shader needs to apply perspective correction:

precision mediump float;
uniform samplerExternalOES u_CameraColorTexture;
in vec3 v_CameraTexCoord;
layout(location = 0) out vec4 o_FragColor;
void main() {
  vec3 tc = (v_CameraTexCoord / v_CameraTexCoord.z);
  o_FragColor = texture(u_CameraColorTexture, tc.xy);
}

See the hello_eis_kotlin sample app for more details.