Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.
How video and AR data are recorded for playback
ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.
Recorded data
It's a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.
Data accessible through MP4-compatible video player
ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.
Primary video track (CPU image track)
ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don't select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.
Camera depth map visualization
This is a video file representing the camera's depth map, recorded from the device's hardware depth sensor, such as a time-of-flight sensor (or ToF sensor), and converted to RGB channel values. This video should only be used for preview purposes.
Additional parameters captured by ARCore (not accessible)
ARCore also captures the following parameters, which are not accessible at this time.
Camera metadata
All video tracks are followed by data tracks containing metadata for each recorded frame.
Depth sensor measurement (raw depth data)
When a camera config with a supported and enabled hardware depth sensor is selected, ARCore records this as a metadata track using information from its hardware depth sensor, such as a time-of-flight sensor (or ToF sensor).
API call events
ARCore records select API calls and callbacks.
IMU sensor measurements
ARCore records measurements from the device's gyrometer and accelerometer sensors.
Other data captured by ARCore
ARCore also records other data, some of which may be sensitive:
- Dataset format versions
- ARCore SDK version
- Google Play Services for AR version
- Device fingerprint (the output of
adb shell getprop ro.build.fingerprint
) - Additional information about sensors used for AR tracking
Prerequisites
Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.
Recording
Start, stop, and check the status of an ARCore session recording.
Record an ARCore session
To record an entire ARCore session, call
ARRecordingManager.StartRecording()
.
This method will fail if a recording is already in progress.
Recording may continue even if the session is paused. If this happens, ARCore
will continue to capture sensor data, while recording the camera feed as a
black screen. To get a recording to stop automatically when the session is
paused, set ARRecordingConfig.AutoStopOnPause
to
true
.
ARCoreRecordingConfig recordingConfig = ScriptableObject.CreateInstance<ARCoreRecordingConfig>();
recordingConfig.Mp4DatasetFilepath = "path/to/file.mp4";
ARRecordingManager.StartRecording(recordingConfig);
Stop a recording
To stop recording, call ARRecordingManager.StopRecording()
.
ARRecordingManager.StopRecording();
Check recording status
ARRecordingManager.RecordingStatus
can be used at any time to determine the recorder's current state.
Debug.Log("Current Recording Status: " + ARRecordingManager.RecordingStatus);
Playback
Playback a previously recorded session
Known issue and workaround
There is a known issue where calls to ARPlaybackManager.SetPlaybackDataset()
returns ErrorPlaybackFailed
.
This happens because it can take several frames for a session to be paused. If
ARPlaybackManager.SetPlaybackDataset()
is called before the session has been paused, it will not be able to access the
session, so it will return an error.
The following code can be used as a workaround solution.
// Workaround for known issue where `ARPlaybackManager.SetPlaybackDataset()`
// returns `ErrorPlaybackFailed` because it can take several frames for a
// session to be paused.
void PlaybackDataset()
{
setPlaybackDataset = true;
// Pause the current session.
session.enabled = false;
// Set a timeout for retrying playback retrieval.
timeout = 10f;
}
// Next frame
void Update()
{
...
if (setPlaybackDataset)
{
PlaybackResult result = ARPlaybackManager.SetPlaybackDataset("path/to/file.mp4");
if (result == PlaybackResult.ErrorPlaybackFailed)
{
// Try to set the dataset again in the next frame.
timeout -= Time.deltaTime;
}
else
{
// Do not set the timeout if the result is something other than ErrorPlaybackFailed.
timeout = -1f;
}
if (timeout < 0.0f)
{
setPlaybackDataset = false;
// If playback is successful, proceed as usual.
// If playback is not successful, handle the error appropriately.
}
}
...
}
To playback a previously recorded session, call ARPlaybackManager.SetPlaybackDataset()
and
provide a filepath for the dataset you wish to playback. You must pause the
session to use this method. Resume the session for the change to take effect.
// Pause the current session when providing the path.
session.enabled = false;
// In the next frame, provide a filepath for the dataset you wish to playback.
ARPlaybackManager.SetPlaybackDataset("path/to/file.mp4");
// In the frame after that, resume the session.
session.enabled = true;
Stop a playback
To stop a playback, call ARPlaybackManager.SetPlaybackDataset()
and set the dataset filepath to
null
.
// The following steps must be completed in separate frames.
session.enabled = false;
ARPlaybackManager.SetPlaybackDataset(null);
session.enabled = true;
Restart playback from the beginning
To restart a playback from the beginning of the dataset,
call ARPlaybackManager.SetPlaybackDataset()
and specify the same MP4 recording before resuming
the session.
// Pause the current session when re-setting the path.
session.enabled = false;
// In the next frame, specify the same dataset.
ARPlaybackManager.SetPlaybackDataset(filepath); // filepath is the same path you used before
// In the frame after that, resume playback from the beginning of the dataset.
session.enabled = true;
Playback a different session
To playback a different dataset, pause the session and call
ARPlaybackManager.SetPlaybackDataset()
, specifying the new dataset before resuming the session.
// Pause the current session when re-setting the path.
session.enabled = false;
// In the next frame, specify a new dataset.
ARPlaybackManager.SetPlaybackDataset(new_filepath); // new_filepath is a different path than the one you used before
// In the frame after that, resume playback from the beginning of the new dataset.
session.enabled = true;
Check playback status
ARPlaybackManager.PlaybackStatus
can be used at any time to determine the current playback status.
Debug.Log("Current Playback Status: " + ARPlaybackManager.PlaybackStatus);