Shared camera access with ARCore

This developer guide walks you through the steps of enabling your app to switch seamlessly between exclusive control of the camera via the Android Camera2 API and sharing camera access with ARCore.

This topic assumes you:

Have completed the ARCore Quickstart
Are familiar with the Android Camera2 API (review the Android-specific Camera2 sample to learn more)

Build and run the sample app

When you build and run the Shared Camera Java sample app, it creates an ARCore session that supports shared camera access. The app starts in non-AR mode,with ARCore paused.

When the app operates in non-AR mode, the camera viewer displays a sepia color effect. When switching to AR mode, the sepia effect switches off as the app returns camera control to ARCore by resuming the paused session.

You can use the AR switch in the app to change modes. During preview, both modes display the number of continuous frames captured by Camera2.

To build and run the Shared Camera Java sample app:

Download and extract the Google ARCore SDK for Android.
Open the samples/shared_camera_java project.
Make sure that your Android device is connected to the development machine via USB. See ARCore Supported devices for detailed information.
In Android Studio, click Run .
Choose your device as the deployment target, and click OK to launch the sample app on your device.
On the device, confirm that you want to allow the app to take pictures and record video.
If prompted to do so, update or install the latest version of ARCore.
Use the AR switch to change between non-AR and AR modes.

Overview of enabling an app to share camera access with ARCore

Follow these steps to implement shared camera access with ARCore in your app. The complete sample code described below is available in the SharedCameraActivity.java within the shared_camera_java sample.

Request `CAMERA` permission

// Verify CAMERA_PERMISSION has been granted.
if (!CameraPermissionHelper.hasCameraPermission(this)) {
    CameraPermissionHelper.requestCameraPermission(this);
}

Ensure ARCore is installed and up to date

// Make sure ARCore is installed and supported on this device.
ArCoreApk.Availability availability =
    ArCoreApk.getInstance().checkAvailability(this);

// Request ARCore installation or update if needed.
switch (availability) {
    case SUPPORTED_INSTALLED:
        break;
    case SUPPORTED_APK_TOO_OLD:
    case SUPPORTED_NOT_INSTALLED:
        ArCoreApk.InstallStatus installStatus =
            ArCoreApk.getInstance().requestInstall(this, …);
        break;
    …
}

Create an ARCore session that supports camera sharing

This involves creating the session, and storing the reference and ID of ARCore shared camera:

// Create ARCore session that supports camera sharing.
sharedSession = new Session(this, EnumSet.of(Session.Feature.SHARED_CAMERA))

// Store the ARCore shared camera reference.
sharedCamera = sharedSession.getSharedCamera();

// Store the ID of the camera used by ARCore.
cameraId = sharedSession.getCameraConfig().getCameraId();

(Optional) Inform ARCore of any custom surfaces

Requesting additional custom surfaces increases the performance demands of the device. To ensure your app performs well, test your app on the devices that your users will use.

By default ARCore will request two streams:

1x YUV CPU stream, currently always 640x480
ARCore uses this stream for motion tracking
1x GPU stream, typically 1920x1080
Use Session#getCameraConfig() to determine the actual GPU stream resolution

On supported devices, the resolution of the GPU stream can be changed by using getSupportedCameraConfigs (CameraConfigFilter cameraConfigFilter) and setCameraConfig (CameraConfig cameraConfig).

As rough indicator, you can expect:

Type of device Simultaneous streams supported

High-end phones

Type of device	Simultaneous streams supported
High-end phones	2x YUV CPU streams, e.g. `640x480` and `1920x1080` 1x GPU stream, e.g. `1920x1080` 1x occasional high res still image (JPEG), e.g. `12MP`
Mid-tier phones	2x YUV CPU streams, e.g. `640x480` and `1920x1080` 1x GPU stream, e.g. `1920x1080` –or– 1x YUV CPU streams, e.g. `640x480` –or– `1920x1080` 1x GPU stream, e.g. `1920x1080` 1x occasional high res still image (JPEG), e.g. `12MP`

2x YUV CPU streams, e.g. 640x480 and 1920x1080
1x GPU stream, e.g. 1920x1080
1x occasional high res still image (JPEG), e.g. 12MP

Mid-tier phones

2x YUV CPU streams, e.g. 640x480 and 1920x1080
1x GPU stream, e.g. 1920x1080

–or–

1x YUV CPU streams, e.g. 640x480 –or– 1920x1080
1x GPU stream, e.g. 1920x1080
1x occasional high res still image (JPEG), e.g. 12MP

To use custom surfaces, such as a CPU image reader surface, call the shared camera's setAppSurfaces(…) method to tell ARCore about the additional surfaces that need to be updated.

sharedCamera.setAppSurfaces(this.cameraId, Arrays.asList(cpuImageReader.getSurface()));

Open the camera

Open the camera using an ARCore-wrapped callback:

// Wrap our callback in a shared camera callback.
CameraDevice.StateCallback wrappedCallback =
    sharedCamera.createARDeviceStateCallback(cameraDeviceCallback, backgroundHandler);

// Store a reference to the camera system service.
cameraManager = (CameraManager) this.getSystemService(Context.CAMERA_SERVICE);

// Open the camera device using the ARCore wrapped callback.
cameraManager.openCamera(cameraId, wrappedCallback, backgroundHandler);

Use the camera device state callback

In the camera device state callback, store a reference to the camera device, and start a new capture session.

public void onOpened(@NonNull CameraDevice cameraDevice) {
    Log.d(TAG, "Camera device ID " + cameraDevice.getId() + " opened.");
    SharedCameraActivity.this.cameraDevice = cameraDevice;
    createCameraPreviewSession();
}

Create a new capture session

Build a new capture request. When doing so, use TEMPLATE_RECORD to ensure that the capture request is compatible with ARCore, and to allow seamless switching between non-AR and AR mode at runtime.

private void createCameraPreviewSession() {
  try {
    …

    // Create an ARCore compatible capture request using `TEMPLATE_RECORD`.
    previewCaptureRequestBuilder =
        cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);

    // Build surfaces list, starting with ARCore provided surfaces.
    List<Surface> surfaceList = sharedCamera.getArCoreSurfaces();

    // (Optional) Add a CPU image reader surface.
    surfaceList.add(cpuImageReader.getSurface());

    // Surface list should now contain three surfaces:
    // 0. sharedCamera.getSurfaceTexture()
    // 1. …
    // 2. cpuImageReader.getSurface()

    // Add ARCore surfaces and CPU image surface targets.
    for (Surface surface : surfaceList) {
      previewCaptureRequestBuilder.addTarget(surface);
    }

    // Wrap our callback in a shared camera callback.
    CameraCaptureSession.StateCallback wrappedCallback =
        sharedCamera.createARSessionStateCallback(cameraSessionStateCallback, backgroundHandler);

    // Create camera capture session for camera preview using ARCore wrapped callback.
    cameraDevice.createCaptureSession(surfaceList, wrappedCallback, backgroundHandler);
  } catch (CameraAccessException e) {
    Log.e(TAG, "CameraAccessException", e);
  }
}

Start in non-AR or AR mode

To begin capturing frames, call captureSession.setRepeatingRequest() from the camera capture session onConfigured() state callback. To start in AR mode, resume the ARCore session within the onActive() callback.

// Repeating camera capture session state callback.
CameraCaptureSession.StateCallback cameraSessionStateCallback =
    new CameraCaptureSession.StateCallback() {

      // Called when the camera capture session is first configured after
      // the app is initialized, and again each time the activity resumes.
      @Override
      public void onConfigured(@NonNull CameraCaptureSession session) {
        captureSession = session;
        setRepeatingCaptureRequest();
      }

      @Override
      public void onActive(@NonNull CameraCaptureSession session) {
        if (arMode && !arcoreActive) {
          resumeARCore();
        }
      }
    };

// Repeating camera capture session capture callback.
private final CameraCaptureSession.CaptureCallback cameraCaptureCallback =
    new CameraCaptureSession.CaptureCallback() {
      …

      @Override
      public void onCaptureCompleted(…) {
        shouldUpdateSurfaceTexture.set(true);
      }
    };

private void setRepeatingCaptureRequest() {
    captureSession.setRepeatingRequest(
        previewCaptureRequestBuilder.build(), cameraCaptureCallback, backgroundHandler);
}

private void resumeARCore() {
    // Resume ARCore.
    sharedSession.resume();
    arcoreActive = true;

    // Set capture session callback while in AR mode.
    sharedCamera.setCaptureCallback(cameraCaptureCallback, backgroundHandler);
}

Switch seamlessly between non-AR or AR modes at runtime

To switch from non-AR to AR mode and resume a paused ARCore session:

// Resume ARCore session.
resumeARCore();

To switch from AR-mode to non-AR mode:

// Pause ARCore.
sharedSession.pause();

// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest();