This developer guide walks you through the steps of enabling your app to switch
seamlessly between exclusive control of the camera via the
Android Camera2
API
and sharing camera access with ARCore.
This topic assumes you:
Have completed the ARCore Quickstart
Are familiar with the Android
Camera2
API (review the Android-specific Camera2 sample to learn more)
Build and run the sample app
When you build and run the Shared Camera Java sample app, it creates an ARCore session that supports shared camera access. The app starts in non-AR mode,with ARCore paused.
When the app operates in non-AR mode, the camera viewer displays a sepia color effect. When switching to AR mode, the sepia effect switches off as the app returns camera control to ARCore by resuming the paused session.
You can use the AR switch in the app to change modes. During preview, both modes
display the number of continuous frames captured by Camera2
.
To build and run the Shared Camera Java sample app:
Download and extract the Google ARCore SDK for Android.
Open the
samples/shared_camera_java
project.Make sure that your Android device is connected to the development machine via USB. See ARCore Supported devices for detailed information.
In Android Studio, click Run
.
Choose your device as the deployment target, and click OK to launch the sample app on your device.
On the device, confirm that you want to allow the app to take pictures and record video.
If prompted to do so, update or install the latest version of ARCore.
Use the AR switch to change between non-AR and AR modes.
Overview of enabling an app to share camera access with ARCore
Follow these steps to implement shared camera access with ARCore in your app.
The complete sample code described below is available in the
SharedCameraActivity.java
within the shared_camera_java
sample.
Request CAMERA
permission
// Verify CAMERA_PERMISSION has been granted.
if (!CameraPermissionHelper.hasCameraPermission(this)) {
CameraPermissionHelper.requestCameraPermission(this);
}
Ensure ARCore is installed and up to date
// Make sure ARCore is installed and supported on this device.
ArCoreApk.Availability availability =
ArCoreApk.getInstance().checkAvailability(this);
// Request ARCore installation or update if needed.
switch (availability) {
case SUPPORTED_INSTALLED:
break;
case SUPPORTED_APK_TOO_OLD:
case SUPPORTED_NOT_INSTALLED:
ArCoreApk.InstallStatus installStatus =
ArCoreApk.getInstance().requestInstall(this, …);
break;
…
}
Create an ARCore session that supports camera sharing
This involves creating the session, and storing the reference and ID of ARCore shared camera:
// Create ARCore session that supports camera sharing.
sharedSession = new Session(this, EnumSet.of(Session.Feature.SHARED_CAMERA))
// Store the ARCore shared camera reference.
sharedCamera = sharedSession.getSharedCamera();
// Store the ID of the camera used by ARCore.
cameraId = sharedSession.getCameraConfig().getCameraId();
(Optional) Inform ARCore of any custom surfaces
Requesting additional custom surfaces increases the performance demands of the device. To ensure your app performs well, test your app on the devices that your users will use.
By default ARCore will request two streams:
- 1x YUV CPU stream, currently always
640x480
ARCore uses this stream for motion tracking - 1x GPU stream, typically
1920x1080
UseSession#getCameraConfig()
to determine the actual GPU stream resolution
On supported devices, the resolution of the GPU stream can be changed by using
getSupportedCameraConfigs (CameraConfigFilter cameraConfigFilter)
and
setCameraConfig (CameraConfig cameraConfig)
.
As rough indicator, you can expect:
Type of device | Simultaneous streams supported |
---|---|
High-end phones |
|
Mid-tier phones |
|
To use custom surfaces, such as a CPU image reader surface, call the
shared camera's setAppSurfaces(…)
method to tell ARCore about the additional
surfaces that need to be updated.
sharedCamera.setAppSurfaces(this.cameraId, Arrays.asList(cpuImageReader.getSurface()));
Open the camera
Open the camera using an ARCore-wrapped callback:
// Wrap our callback in a shared camera callback.
CameraDevice.StateCallback wrappedCallback =
sharedCamera.createARDeviceStateCallback(cameraDeviceCallback, backgroundHandler);
// Store a reference to the camera system service.
cameraManager = (CameraManager) this.getSystemService(Context.CAMERA_SERVICE);
// Open the camera device using the ARCore wrapped callback.
cameraManager.openCamera(cameraId, wrappedCallback, backgroundHandler);
Use the camera device state callback
In the camera device state callback, store a reference to the camera device, and start a new capture session.
public void onOpened(@NonNull CameraDevice cameraDevice) {
Log.d(TAG, "Camera device ID " + cameraDevice.getId() + " opened.");
SharedCameraActivity.this.cameraDevice = cameraDevice;
createCameraPreviewSession();
}
Create a new capture session
Build a new capture request. When doing so, use TEMPLATE_RECORD
to ensure
that the capture request is compatible with ARCore, and to allow seamless
switching between non-AR and AR mode at runtime.
private void createCameraPreviewSession() {
try {
…
// Create an ARCore compatible capture request using `TEMPLATE_RECORD`.
previewCaptureRequestBuilder =
cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);
// Build surfaces list, starting with ARCore provided surfaces.
List<Surface> surfaceList = sharedCamera.getArCoreSurfaces();
// (Optional) Add a CPU image reader surface.
surfaceList.add(cpuImageReader.getSurface());
// Surface list should now contain three surfaces:
// 0. sharedCamera.getSurfaceTexture()
// 1. …
// 2. cpuImageReader.getSurface()
// Add ARCore surfaces and CPU image surface targets.
for (Surface surface : surfaceList) {
previewCaptureRequestBuilder.addTarget(surface);
}
// Wrap our callback in a shared camera callback.
CameraCaptureSession.StateCallback wrappedCallback =
sharedCamera.createARSessionStateCallback(cameraSessionStateCallback, backgroundHandler);
// Create camera capture session for camera preview using ARCore wrapped callback.
cameraDevice.createCaptureSession(surfaceList, wrappedCallback, backgroundHandler);
} catch (CameraAccessException e) {
Log.e(TAG, "CameraAccessException", e);
}
}
Start in non-AR or AR mode
To begin capturing frames, call captureSession.setRepeatingRequest()
from the
camera capture session onConfigured()
state callback. To start in AR mode,
resume the ARCore session within the onActive()
callback.
// Repeating camera capture session state callback.
CameraCaptureSession.StateCallback cameraSessionStateCallback =
new CameraCaptureSession.StateCallback() {
// Called when the camera capture session is first configured after
// the app is initialized, and again each time the activity resumes.
@Override
public void onConfigured(@NonNull CameraCaptureSession session) {
captureSession = session;
setRepeatingCaptureRequest();
}
@Override
public void onActive(@NonNull CameraCaptureSession session) {
if (arMode && !arcoreActive) {
resumeARCore();
}
}
};
// Repeating camera capture session capture callback.
private final CameraCaptureSession.CaptureCallback cameraCaptureCallback =
new CameraCaptureSession.CaptureCallback() {
…
@Override
public void onCaptureCompleted(…) {
shouldUpdateSurfaceTexture.set(true);
}
};
private void setRepeatingCaptureRequest() {
captureSession.setRepeatingRequest(
previewCaptureRequestBuilder.build(), cameraCaptureCallback, backgroundHandler);
}
private void resumeARCore() {
// Resume ARCore.
sharedSession.resume();
arcoreActive = true;
// Set capture session callback while in AR mode.
sharedCamera.setCaptureCallback(cameraCaptureCallback, backgroundHandler);
}
Switch seamlessly between non-AR or AR modes at runtime
To switch from non-AR to AR mode and resume a paused ARCore session:
// Resume ARCore session.
resumeARCore();
To switch from AR-mode to non-AR mode:
// Pause ARCore.
sharedSession.pause();
// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest();