SharedCamera

public class SharedCamera

The SharedCamera class allows the app to share control of the camera with ARCore and obtain image readers and surfaces for rendering. Images can be read concurrently while ARCore is running or while ARCore is paused.

Use this class to obtain wrappers for the CameraDevice and CameraCaptureSession state callbacks, so that camera events and data objects can be shared with ARCore.

When a session is created for shared camera access, ARCore's behavior changes in the following ways:

  • While ARCore is active, the app can use Camera2 APIs, except it must not call CameraCaptureSession#setRepeatingRequest as this will interfere with ARCore operation. The Camera2 APIs can be used directly without restriction while ARCore is paused.
  • Session.getSharedCamera() can be used to obtain an instance of the SharedCamera object.
  • Although ARCore will continue to track normally, it will not be able to use the depth sensor even if one is present on the device.

Example of app activity using shared camera: 

Session sharedSession;
 SharedCamera sharedCamera;
 void onCreate(Bundle savedInstanceState) {
   sharedSession = new Session(context, EnumSet.of(Session.Feature.SHARED_CAMERA));
   openCameraForSharing();
   createCameraCaptureSession();
   // ...
 }
 private void openCameraForSharing() {
   sharedCamera = sharedSession.getSharedCamera();
   // Use callback wrapper.
   CameraManager.openCamera(
       cameraID,
       sharedCamera.createARDeviceStateCallback(appDeviceStateCallback, appHandler),
       appHandler);
 }
 private void createCameraCaptureSession() {
   // Get list of ARCore created surfaces. Required for ARCore tracking.
   ArrayList<Surface> surfaceList = sharedCamera.getSurfaces();
   // (Optional) Add a custom CPU image reader surface on devices that support CPU image access.
   ImageReader cpuImageReader = ImageReader.newInstance(…);
   surfaceList.add(cpuImageReader.getSurface());
   // Use callback wrapper.
   cameraDevice.createCaptureSession(
       surfaceList,
       sharedCamera.createARSessionStateCallback(
           appSessionStateCallback, appHandler),
       appHandler);
 }

Public Methods

CameraDevice.StateCallback
createARDeviceStateCallback(CameraDevice.StateCallback appCallback, Handler appHandler)
The openCamera command on the CameraManager should use this wrapped callback as input.
CameraCaptureSession.StateCallback
createARSessionStateCallback(CameraCaptureSession.StateCallback appCallback, Handler appHandler)
Wraps a CameraCaptureSession.StateCallback to allow ARCore to control the camera.
List<Surface>
getArCoreSurfaces()
Gets access to the list of surfaces created and used by ARCore.
SurfaceTexture
getSurfaceTexture()
Gets access to the SurfaceTexture provided by ARCore.
void
setAppSurfaces(String cameraId, List<Surface> surfaces)
Set app created surfaces, to receive additional images when ARCore is active.
void
setCaptureCallback(CameraCaptureSession.CaptureCallback appCaptureCallback, Handler appHandler)
Set CameraCaptureSession.CaptureCallback by sending it to ARCore in order to provide metadata information to the client.

Inherited Methods

Public Methods

createARDeviceStateCallback

public CameraDevice.StateCallback createARDeviceStateCallback(
  CameraDevice.StateCallback appCallback,
  Handler appHandler
)

The openCamera command on the CameraManager should use this wrapped callback as input. 

CameraManager.openCamera(
   cameraID,
   sharedCamera.createARDeviceStateCallback(appDeviceStateCallback, appHandler),
   appHandler);
 }
 See SharedCamera class level documentation for a more complete example.

Details
Parameters
appCallback
appHandler

createARSessionStateCallback

public CameraCaptureSession.StateCallback createARSessionStateCallback(
  CameraCaptureSession.StateCallback appCallback,
  Handler appHandler
)

Wraps a CameraCaptureSession.StateCallback to allow ARCore to control the camera. The result of this call should be used in CameraDevice#createCaptureSession as follows: 

cameraDevice.createCaptureSession(
   surfaceList,
   sharedCamera.createARSessionStateCallback(appSessionStateCallback, appHandler),
   appHandler);
See SharedCamera class level documentation for a more complete example.

Details
Parameters
appCallback
appHandler

getArCoreSurfaces

public List<Surface> getArCoreSurfaces()

Gets access to the list of surfaces created and used by ARCore.

Multiple calls to this function will return the same list.

ARCore always creates two surfaces. A third surface is optionally created if requested by the app by selecting a non-default camera config. The order of surfaces in the returned list is guaranteed to be:

  1. GPU texture surface (best available resolution, typically 1920x1280)
  2. Motion tracking surface (640x480 resolution)
  3. (Optional) Higher resolution CPU image, if set by the app using Session.setCameraConfig(CameraConfig)

Note, when calling CameraCaptureSession.setRepeatingRequest(android.hardware.camera2.CaptureRequest, android.hardware.camera2.CameraCaptureSession.CaptureCallback, android.os.Handler), you must pass in a complete list of surfaces that includes:

Details
Returns list of the surfaces created by ARCore. Does not include additional surfaces created by the app.

getSurfaceTexture

public SurfaceTexture getSurfaceTexture()

Gets access to the SurfaceTexture provided by ARCore.

This method should be called after the camera has been successfully opened, for example in the onOpened callback in android.hardware.camera2.CameraDevice.StateCallback.

ARCore will not attach the SurfaceTexture directly to the textures from Session.setCameraTextureName(int) or Session.setCameraTextureNames(int[]) if frame buffering is enabled. The app must detach and reattach to the desired texture in order to continue using this object while ARCore is paused.

setAppSurfaces

public void setAppSurfaces(
  String cameraId,
  List<Surface> surfaces
)

Set app created surfaces, to receive additional images when ARCore is active.

It is the app's responsibility to ensure that the CameraDevice and ARCore both support the full set of streams/surfaces/resolutions, those automatically created by ARCore and returned by getArCoreSurfaces(), and the app created surfaces specified here.

A single additional surface, with a resolution equal to one of the CPU resolutions returned by Session.getSupportedCameraConfigs(CameraConfigFilter), is guaranteed to be supported by the CameraDevice and ARCore. Additional surfaces beyond one, and resolutions not listed in Session.getSupportedCameraConfigs(CameraConfigFilter), might or might not be supported by the device and/or ARCore. It is the app developer's responsibility to verify that ARCore and all targeted devices support the requested app surfaces.

The provided list must only include any surfaces that are in addition to the ones returned by getArCoreSurfaces().

Note, when calling CameraCaptureSession.setRepeatingRequest(android.hardware.camera2.CaptureRequest, android.hardware.camera2.CameraCaptureSession.CaptureCallback, android.os.Handler), you must pass in a complete list of surfaces that includes:

Details
Parameters
cameraId the ID of this camera device.
surfaces the list of a Surface.

setCaptureCallback

public void setCaptureCallback(
  CameraCaptureSession.CaptureCallback appCaptureCallback,
  Handler appHandler
)

Set CameraCaptureSession.CaptureCallback by sending it to ARCore in order to provide metadata information to the client.

Details
Parameters
appCaptureCallback
appHandler