Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.
How video and AR data are recorded for playback
ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.
It's a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.
Data accessible through MP4-compatible video player
ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.
Primary video track (CPU image track)
ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don't select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.
Camera depth map visualization
This is a video file representing the camera's depth map, recorded from the device's hardware depth sensor, such as a time-of-flight sensor (or ToF sensor), and converted to RGB channel values. This video should only be used for preview purposes.
Additional parameters captured by ARCore (not accessible)
ARCore also captures the following parameters, which are not accessible at this time.
All video tracks are followed by data tracks containing metadata for each recorded frame.
Depth sensor measurement (raw depth data)
When a camera config with a supported and enabled hardware depth sensor is selected, ARCore records this as a metadata track using information from its hardware depth sensor, such as a time-of-flight sensor (or ToF sensor).
API call events
ARCore records select API calls and callbacks.
IMU sensor measurements
ARCore records measurements from the device's gyrometer and accelerometer sensors.
Other data captured by ARCore
ARCore also records other data, some of which may be sensitive:
- Dataset format versions
- ARCore SDK version
- Google Play Services for AR version
- Device fingerprint (the output of
adb shell getprop ro.build.fingerprint)
- Additional information about sensors used for AR tracking
Start, stop, and check the status of an ARCore session recording.
Record an ARCore session
To record an entire ARCore session, call
before the first call to
ArSession_resume(ArSession*). Recording automatically
starts when the session resumes. To automatically stop recording when the
session is paused, call
ArRecordingConfig_setAutoStopOnPause(ArSession*, ArRecordingConfig*). To record a partial session, call
while the session is running.
ArRecordingConfig* recording_config = nullptr; ArRecordingConfig_create(ar_session, &recording_config); ArRecordingConfig_setMp4DatasetFilePath(ar_session, recording_config, "/sdcard/recording.mp4"); ArRecordingConfig_setAutoStopOnPause(ar_session, recording_config, true); ArStatus status; status = ArSession_startRecording(ar_session, recording_config); … // Resume ARCore session to start recording. status = ArSession_resume(ar_session); … // Recording ends. status = ArSession_pause(ar_session);
Stop a recording
To stop recording without pausing the currently running AR session, call
status = ArSession_stopRecording(ar_session);
Check recording status
can be used at any time to determine the current
ArRecordingStatus recording_status; // Can be called at any time. ArSession_getRecordingStatus(ar_session, &recording_status);
Play back previously recorded AR sessions.
Play back a previously recorded session
To play back a previously recorded session, call
before the first call to
// Specify previously recorded MP4 file. ArStatus status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording.mp4"); … // Playback starts from the beginning of the dataset. status = ArSession_resume(ar_session); … // Pause AR session, but allow playback to silently continue. status = ArSession_pause(ar_session); … // Resume AR session. Playback continues with gap to paused session. status = ArSession_resume(ar_session);
Restart playback from the beginning
To restart a playback from the beginning of the dataset, pause the session and
specifying the same MP4 recording, before resuming the session.
status = ArSession_pause(ar_session); // Pause and specify the *same* dataset: ArStatus status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording.mp4"); // Playback starts from the *beginning* of the dataset. status = ArSession_resume(ar_session);
Play back a different session
To play back a different dataset, pause the session and specify the new dataset before resuming the session.
ArStatus status = ArSession_pause(ar_session); // Pause and specify a *different* dataset: status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording2.mp4"); // Playback starts from the *beginning* of the new dataset. status = ArSession_resume(ar_session);
Check playback status
You can use
ArSession_getPlaybackStatus(ArSession*, ArPlaybackStatus*) at any time to determine the current
ArPlaybackStatus playback_status; // Can be called at any time. ArSession_getPlaybackStatus(ar_session, &playback_status);