Recording and Playback developer guide for AR Foundation for Android

Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.

How video and AR data are recorded for playback

ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.

Recorded data

It's a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.

Data accessible through MP4-compatible video player

ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.

Primary video track (CPU image track)

ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don't select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.

Note: ARCore does not capture the GPU texture.

Camera depth map visualization

This is a video file representing the camera's depth map, recorded from the device's hardware depth sensor, such as a time-of-flight sensor (or ToF sensor), and converted to RGB channel values. This video should only be used for preview purposes.

Additional parameters captured by ARCore (not accessible)

ARCore also captures the following parameters, which are not accessible at this time.

Camera metadata

All video tracks are followed by data tracks containing metadata for each recorded frame.

Depth sensor measurement (raw depth data)

When a camera config with a supported and enabled hardware depth sensor is selected, ARCore records this as a metadata track using information from its hardware depth sensor, such as a time-of-flight sensor (or ToF sensor).

API call events

ARCore records select API calls and callbacks.

IMU sensor measurements

ARCore records measurements from the device's gyrometer and accelerometer sensors.

Other data captured by ARCore

ARCore also records other data, some of which may be sensitive:

• Dataset format versions
• ARCore SDK version
• Google Play Services for AR version
• Device fingerprint (the output of adb shell getprop ro.build.fingerprint)
• Additional information about sensors used for AR tracking

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Recording

Start, stop, and check the status of an ARCore session recording.

Record an ARCore session

To record an entire ARCore session, call ARRecordingManager.StartRecording(). This method will fail if a recording is already in progress.

Recording may continue even if the session is paused. If this happens, ARCore will continue to capture sensor data, while recording the camera feed as a black screen. To get a recording to stop automatically when the session is paused, set ARRecordingConfig.AutoStopOnPause to true.

ARCoreRecordingConfig recordingConfig = ScriptableObject.CreateInstance<ARCoreRecordingConfig>();
recordingConfig.Mp4DatasetFilepath = "path/to/file.mp4";

ARRecordingManager.StartRecording(recordingConfig);

Stop a recording

To stop recording, call ARRecordingManager.StopRecording().

ARRecordingManager.StopRecording();

Check recording status

ARRecordingManager.RecordingStatus can be used at any time to determine the recorder's current state.

Debug.Log("Current Recording Status: " + ARRecordingManager.RecordingStatus);

Playback

Playback a previously recorded session

Caution: Due to the way session data is processed, ARCore APIs may produce different results during playback than during recording. They may also produce different results during subsequent playback sessions. For example, the number of detected trackables, the precise timing of their detection, and their poses over time may be different during playback.

Key Point: Once playback has started, pausing the session (by disabling the ARSession) will suspend processing of all camera image frames and any other recorded sensor data in the dataset. Camera image frames and sensor frame data that is discarded in this way will not be reprocessed when the session is again resumed (by re-enabling the ARSession). AR tracking for the session will generally suffer due to the gap in processed data.

Known issue and workaround

There is a known issue where calls to ARPlaybackManager.SetPlaybackDataset() return ErrorPlaybackFailed. This happens because it can take several frames for a session to be paused. If ARPlaybackManager.SetPlaybackDataset() is called before the session has been paused, it will not be able to access the session, so it will return an error.

The following code can be used as a workaround solution.

// Workaround for known issue where `ARPlaybackManager.SetPlaybackDataset()`
// returns `ErrorPlaybackFailed` because it can take several frames for a
// session to be paused.

// Reference to the ARSession component in the scene.
ARSession session;

void PlaybackDataset()
{
    setPlaybackDataset = true;

    // Pause the current AR session.
    session.enabled = false;

    // Set a timeout for retrying playback retrieval.
    timeout = 10f;
}

// Next frame
void Update()
{
    ...

    if (setPlaybackDataset)
    {
        PlaybackResult result = ARPlaybackManager.SetPlaybackDataset("path/to/file.mp4");
        if (result == PlaybackResult.ErrorPlaybackFailed)
        {
            // Try to set the dataset again in the next frame.
            timeout -= Time.deltaTime;
        }
        else
        {
            // Do not set the timeout if the result is something other than ErrorPlaybackFailed.
            timeout = -1f;
        }

        if (timeout < 0.0f)
        {
            setPlaybackDataset = false;
            // If playback is successful, proceed as usual.
            // If playback is not successful, handle the error appropriately.
        }
    }

    ...
}

To play back a previously recorded session, call ARPlaybackManager.SetPlaybackDataset() and provide a file path for the dataset you wish to play back. You must pause the session to use this method. Resume the session for the change to take effect.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, provide a file path for the dataset you wish to play back.
ARPlaybackManager.SetPlaybackDataset("path/to/file.mp4");

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the dataset.
session.enabled = true;

Stop a playback

To stop a playback, call ARPlaybackManager.SetPlaybackDataset() and set the dataset filepath to null.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, unset the playback dataset file path.
ARPlaybackManager.SetPlaybackDataset(null);

// In the frame after that, re-enable the ARSession to resume the session using
// the device camera and other sensors.
session.enabled = true;

Restart playback from the beginning

To restart a playback from the beginning of the dataset, call ARPlaybackManager.SetPlaybackDataset() and specify the same MP4 recording before resuming the session.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, specify the same dataset file path.
ARPlaybackManager.SetPlaybackDataset(filepath); // Same path that was previously set.

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the dataset.
session.enabled = true;

Playback a different session

To play back a different dataset, pause the session and call ARPlaybackManager.SetPlaybackDataset(), specifying the new dataset before resuming the session.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, specify a new dataset file path.
ARPlaybackManager.SetPlaybackDataset(new_filepath); // Different path than was previously set.

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the new dataset.
session.enabled = true;

Check playback status

ARPlaybackManager.PlaybackStatus can be used at any time to determine the current playback status.

Debug.Log("Current Playback Status: " + ARPlaybackManager.PlaybackStatus);