Shared camera access with ARCore

This developer guide walks you through the steps of enabling your app to switch seamlessly between exclusive control of the camera via the Android Camera2 API and sharing camera access with ARCore.

This topic assumes you:

Build and run the sample app

When you build and run the Shared Camera Java sample app, it creates an ARCore session that supports shared camera access. The app starts in non-AR mode,with ARCore paused.

When the app operates in non-AR mode, the camera viewer displays a sepia color effect. When switching to AR mode, the sepia effect switches off as the app returns camera control to ARCore by resuming the paused session.

You can use the AR switch in the app to change modes. During preview, both modes display the number of continuous frames captured by Camera2.

To build and run the Shared Camera Java sample app:

  1. Download and extract the Google ARCore SDK for Android.

  2. Open the samples/shared_camera_java project.

  3. Make sure that your Android device is connected to the development machine via USB. See ARCore Supported devices for detailed information.

  4. In Android Studio, click Run .

  5. Choose your device as the deployment target, and click OK to launch the sample app on your device.

  6. On the device, confirm that you want to allow the app to take pictures and record video.

  7. If prompted to do so, update or install the latest version of ARCore.

  8. Use the AR switch to change between non-AR and AR modes.

Overview of enabling an app to share camera access with ARCore

Follow these steps to implement shared camera access with ARCore in your app. All code snippets are available in the SharedCameraActivity.java within the shared_camera_java sample.

Request CAMERA permission

In order to be able to use the device's camera, the user must grant your app the CAMERA permission. The ARCore samples include a CameraPermissionHelper, which provides utilities to request the correct permission for your app.

Java

protected void onResume() {
  // Request the camera permission, if necessary.
  if (!CameraPermissionHelper.hasCameraPermission(this)) {
      CameraPermissionHelper.requestCameraPermission(this);
  }
}

Kotlin

override fun onResume() {
  // Request the camera permission, if necessary.
  if (!CameraPermissionHelper.hasCameraPermission(this)) {
    CameraPermissionHelper.requestCameraPermission(this)
  }
}

Ensure ARCore is installed and up to date

ARCore must be installed and up-to-date before it can be used. The following snippet shows how to request an installation of ARCore if it has not already been installed on the device.

Java

boolean isARCoreSupportedAndUpToDate() {
  // Make sure that ARCore is installed and supported on this device.
  ArCoreApk.Availability availability = ArCoreApk.getInstance().checkAvailability(this);
  switch (availability) {
    case SUPPORTED_INSTALLED:
      return true;

    case SUPPORTED_APK_TOO_OLD:
    case SUPPORTED_NOT_INSTALLED:
        // Requests an ARCore installation or updates ARCore if needed.
        ArCoreApk.InstallStatus installStatus = ArCoreApk.getInstance().requestInstall(this, userRequestedInstall);
        switch (installStatus) {
          case INSTALL_REQUESTED:
            return false;
          case INSTALLED:
            return true;
        }
      return false;

    default:
      // Handle the error. For example, show the user a snackbar that tells them
      // ARCore is not supported on their device.
      return false;
  }
}

Kotlin

// Determine ARCore installation status.
// Requests an ARCore installation or updates ARCore if needed.
fun isARCoreSupportedAndUpToDate(): Boolean {
  when (ArCoreApk.getInstance().checkAvailability(this)) {
    Availability.SUPPORTED_INSTALLED -> return true

    Availability.SUPPORTED_APK_TOO_OLD,
    Availability.SUPPORTED_NOT_INSTALLED -> {
      when(ArCoreApk.getInstance().requestInstall(this, userRequestedInstall)) {
        InstallStatus.INSTALLED -> return true
        else -> return false
      }
    }

    else -> {
      // Handle the error. For example, show the user a snackbar that tells them
      // ARCore is not supported on their device.
      return false
    }
  }
}

Create an ARCore session that supports camera sharing

This involves creating the session and storing the reference and ID of ARCore shared camera:

Java

// Create an ARCore session that supports camera sharing.
sharedSession = new Session(this, EnumSet.of(Session.Feature.SHARED_CAMERA))

// Store the ARCore shared camera reference.
sharedCamera = sharedSession.getSharedCamera();

// Store the ID of the camera that ARCore uses.
cameraId = sharedSession.getCameraConfig().getCameraId();

Kotlin

// Create an ARCore session that supports camera sharing.
sharedSession = Session(this, EnumSet.of(Session.Feature.SHARED_CAMERA))

// Store the ARCore shared camera reference.
sharedCamera = sharedSession.sharedCamera

// Store the ID of the camera that ARCore uses.
cameraId = sharedSession.cameraConfig.cameraId

(Optional) Inform ARCore of any custom surfaces

Requesting additional custom surfaces increases the performance demands of the device. To ensure it performs well, test your app on the devices that your users will use.

ARCore will request two streams by default:

  1. 1x YUV CPU stream, currently always 640x480.
    ARCore uses this stream for motion tracking.
  2. A 1x GPU stream, typically 1920x1080
    Use Session#getCameraConfig() to determine the current GPU stream resolution.

You can change the resolution of the GPU stream on supported devices by using getSupportedCameraConfigs() and setCameraConfig().

As rough indicator, you can expect:

Type of device Simultaneous streams supported
High-end phones
  • 2x YUV CPU streams, e.g. 640x480 and 1920x1080
  • 1x GPU stream, e.g. 1920x1080
  • 1x occasional high res still image (JPEG), e.g. 12MP
Mid-tier phones
  • 2x YUV CPU streams, e.g. 640x480 and 1920x1080
  • 1x GPU stream, e.g. 1920x1080
–or–
  • 1x YUV CPU streams, e.g. 640x480 –or– 1920x1080
  • 1x GPU stream, e.g. 1920x1080
  • 1x occasional high res still image (JPEG), e.g. 12MP

To use custom surfaces, such as a CPU image reader surface, make sure to add it to the list of surfaces that need to be updated (for example, an ImageReader).

Java

sharedCamera.setAppSurfaces(this.cameraId, Arrays.asList(imageReader.getSurface()));

Kotlin

sharedCamera.setAppSurfaces(this.cameraId, listOf(imageReader.surface))

Open the camera

Open the camera using an ARCore-wrapped callback:

Java

// Wrap the callback in a shared camera callback.
CameraDevice.StateCallback wrappedCallback =
    sharedCamera.createARDeviceStateCallback(cameraDeviceCallback, backgroundHandler);

// Store a reference to the camera system service.
cameraManager = (CameraManager) this.getSystemService(Context.CAMERA_SERVICE);

// Open the camera device using the ARCore wrapped callback.
cameraManager.openCamera(cameraId, wrappedCallback, backgroundHandler);

Kotlin

// Wrap the callback in a shared camera callback.
val wrappedCallback = sharedCamera.createARDeviceStateCallback(cameraDeviceCallback, backgroundHandler)

// Store a reference to the camera system service.
val cameraManager = this.getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Open the camera device using the ARCore wrapped callback.
cameraManager.openCamera(cameraId, wrappedCallback, backgroundHandler)

Use the camera device state callback

In the camera device state callback store a reference to the camera device, and start a new capture session.

Java

public void onOpened(@NonNull CameraDevice cameraDevice) {
    Log.d(TAG, "Camera device ID " + cameraDevice.getId() + " opened.");
    SharedCameraActivity.this.cameraDevice = cameraDevice;
    createCameraPreviewSession();
}

Kotlin

fun onOpened(cameraDevice: CameraDevice) {
  Log.d(TAG, "Camera device ID " + cameraDevice.id + " opened.")
  this.cameraDevice = cameraDevice
  createCameraPreviewSession()
}

Create a new capture session

Build a new capture request. Use TEMPLATE_RECORD to ensure that the capture request is compatible with ARCore, and to allow seamless switching between non-AR and AR mode at runtime.

Java

void createCameraPreviewSession() {
  try {
    // Create an ARCore-compatible capture request using `TEMPLATE_RECORD`.
    previewCaptureRequestBuilder =
        cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);

    // Build a list of surfaces, starting with ARCore provided surfaces.
    List<Surface> surfaceList = sharedCamera.getArCoreSurfaces();

    // (Optional) Add a CPU image reader surface.
    surfaceList.add(cpuImageReader.getSurface());

    // The list should now contain three surfaces:
    // 0. sharedCamera.getSurfaceTexture()
    // 1. …
    // 2. cpuImageReader.getSurface()

    // Add ARCore surfaces and CPU image surface targets.
    for (Surface surface : surfaceList) {
      previewCaptureRequestBuilder.addTarget(surface);
    }

    // Wrap our callback in a shared camera callback.
    CameraCaptureSession.StateCallback wrappedCallback =
        sharedCamera.createARSessionStateCallback(cameraSessionStateCallback, backgroundHandler);

    // Create a camera capture session for camera preview using an ARCore wrapped callback.
    cameraDevice.createCaptureSession(surfaceList, wrappedCallback, backgroundHandler);
  } catch (CameraAccessException e) {
    Log.e(TAG, "CameraAccessException", e);
  }
}

Kotlin

fun createCameraPreviewSession() {
  try {
    // Create an ARCore-compatible capture request using `TEMPLATE_RECORD`.
    previewCaptureRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD)

    // Build a list of surfaces, starting with ARCore provided surfaces.
    val surfaceList: MutableList<Surface> = sharedCamera.arCoreSurfaces

    // (Optional) Add a CPU image reader surface.
    surfaceList.add(cpuImageReader.getSurface())

    // The list should now contain three surfaces:
    // 0. sharedCamera.getSurfaceTexture()
    // 1. …
    // 2. cpuImageReader.getSurface()

    // Add ARCore surfaces and CPU image surface targets.
    for (surface in surfaceList) {
      previewCaptureRequestBuilder.addTarget(surface)
    }

    // Wrap the callback in a shared camera callback.
    val wrappedCallback = sharedCamera.createARSessionStateCallback(cameraSessionStateCallback, backgroundHandler)

    // Create a camera capture session for camera preview using an ARCore wrapped callback.
    cameraDevice.createCaptureSession(surfaceList, wrappedCallback, backgroundHandler)
  } catch (e: CameraAccessException) {
    Log.e(TAG, "CameraAccessException", e)
  }
}

Start in non-AR or AR mode

To begin capturing frames, call captureSession.setRepeatingRequest() from the camera capture session onConfigured() state callback. Resume the ARCore session within the onActive() callback to start in AR mode.

Java

// Repeating camera capture session state callback.
CameraCaptureSession.StateCallback cameraSessionStateCallback =
    new CameraCaptureSession.StateCallback() {

      // Called when ARCore first configures the camera capture session after
      // initializing the app, and again each time the activity resumes.
      @Override
      public void onConfigured(@NonNull CameraCaptureSession session) {
        captureSession = session;
        setRepeatingCaptureRequest();
      }

      @Override
      public void onActive(@NonNull CameraCaptureSession session) {
        if (arMode && !arcoreActive) {
          resumeARCore();
        }
      }
    };

// A repeating camera capture session capture callback.
CameraCaptureSession.CaptureCallback cameraCaptureCallback =
    new CameraCaptureSession.CaptureCallback() {
      @Override
      public void onCaptureCompleted(…) {
        shouldUpdateSurfaceTexture.set(true);
      }
    };

void setRepeatingCaptureRequest() {
    captureSession.setRepeatingRequest(
        previewCaptureRequestBuilder.build(), cameraCaptureCallback, backgroundHandler);
}

void resumeARCore() {
    // Resume ARCore.
    sharedSession.resume();
    arcoreActive = true;

    // Set the capture session callback while in AR mode.
    sharedCamera.setCaptureCallback(cameraCaptureCallback, backgroundHandler);
}

Kotlin

val cameraSessionStateCallback = object : CameraCaptureSession.StateCallback() {
      // Called when ARCore first configures the camera capture session after
      // initializing the app, and again each time the activity resumes.
  override fun onConfigured(session: CameraCaptureSession) {
    captureSession = session
    setRepeatingCaptureRequest()
  }

  override fun onActive(session: CameraCaptureSession) {
    if (arMode && !arcoreActive) {
      resumeARCore()
    }
  }
}

val cameraCaptureCallback = object : CameraCaptureSession.CaptureCallback() {
  override fun onCaptureCompleted(
    session: CameraCaptureSession,
    request: CaptureRequest,
    result: TotalCaptureResult
  ) {
    shouldUpdateSurfaceTexture.set(true);
  }
}

fun setRepeatingCaptureRequest() {
  captureSession.setRepeatingRequest(
    previewCaptureRequestBuilder.build(), cameraCaptureCallback, backgroundHandler
  )
}

fun resumeARCore() {
    // Resume ARCore.
    sharedSession.resume()
    arcoreActive = true

    // Set the capture session callback while in AR mode.
    sharedCamera.setCaptureCallback(cameraCaptureCallback, backgroundHandler)
}

Switch seamlessly between non-AR or AR modes at runtime

To switch from non-AR to AR mode and resume a paused ARCore session:

Java

// Resume the ARCore session.
resumeARCore();

Kotlin

// Resume the ARCore session.
resumeARCore()

To switch from AR-mode to non-AR mode:

Java

// Pause ARCore.
sharedSession.pause();

// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest();

Kotlin

// Pause ARCore.
sharedSession.pause()

// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest()