Google is committed to advancing racial equity for Black communities. See how.

Recording and Playback developer guide for Android NDK

Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.

How video and AR data are recorded for playback

ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.

Recorded data

It’s a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.

Data accessible through MP4-compatible video player

ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.

Primary video track (CPU image track)

ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don’t select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.

Camera depth map visualization

This is a video file representing the camera’s depth map, recorded from the device’s time-of-flight (ToF) sensors, and converted to RGB channel values. This video should only be used for preview purposes.

Additional parameters captured by ARCore (not accessible)

ARCore also captures the following parameters, which are not accessible at this time.

Camera metadata

All video tracks are followed by data tracks containing metadata for each recorded frame.

Depth sensor measurement (raw depth data)

When a camera config with an enabled active depth sensor is selected, ARCore records this as a metadata track using information from its ToF sensor.

API call events

ARCore records select API calls and callbacks.

IMU sensor measurements

ARCore records measurements from the device’s gyrometer and accelerometer sensors.

Other data captured by ARCore

ARCore also records other data, some of which may be sensitive:

  • Dataset format versions
  • ARCore SDK version
  • Google Play Services for AR version
  • Device fingerprint (the output of adb shell getprop ro.build.fingerprint)
  • Additional information about sensors used for AR tracking

Recording

Start, stop, and check the status of an ARCore session recording.

Record an ARCore session

To record an entire ARCore session, call ArSession_startRecording(ArSession*, ArRecordingConfig*) before the first call to ArSession_resume(ArSession*). Recording automatically starts when the session resumes. To automatically stop recording when the session is paused, call ArRecordingConfig_setAutoStopOnPause(ArSession*, ArRecordingConfig*). To record a partial session, call ArSession_startRecording(ArSession*, ArRecordingConfig*) while the session is running.

ArRecordingConfig* recording_config = nullptr;
ArRecordingConfig_create(ar_session, &recording_config);
ArRecordingConfig_setMp4DatasetFilePath(ar_session, recording_config, "/sdcard/recording.mp4");
ArRecordingConfig_setAutoStopOnPause(ar_session, recording_config, true);

ArStatus status;
status = ArSession_startRecording(ar_session, recording_config);
…
// Resume ARCore session to start recording.
status = ArSession_resume(ar_session);
…
// Recording ends.
status = ArSession_pause(ar_session);

Stop a recording

To stop recording without pausing the currently running AR session, call ArSession_stopRecording(ArSession*).

status = ArSession_stopRecording(ar_session);

Check recording status

ArSession_getRecordingStatus(ArSession*, ArRecordingStatus*) can be used at any time to determine the current ArRecordingStatus.

ArRecordingStatus recording_status;
// Can be called at any time.
ArSession_getRecordingStatus(ar_session, &recording_status);

Playback

Play back previously recorded AR sessions.

Play back a previously recorded session

To play back a previously recorded session, call ArSession_setPlaybackDataset(ArSession*, char*) before the first call to ArSession_resume(ArSession*).

// Specify previously recorded MP4 file.
ArStatus status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording.mp4");
…
// Playback starts from the beginning of the dataset.
status = ArSession_resume(ar_session);
…
// Pause AR session, but allow playback to silently continue.
status = ArSession_pause(ar_session);
…
// Resume AR session. Playback continues with gap to paused session.
status = ArSession_resume(ar_session);

Restart playback from the beginning

To restart a playback from the beginning of the dataset, pause the session and call ArSession_setPlaybackDataset(ArSession*, char*), specifying the same MP4 recording, before resuming the session.

status = ArSession_pause(ar_session);
// Pause and specify the *same* dataset:
ArStatus status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording.mp4");
// Playback starts from the *beginning* of the dataset.
status = ArSession_resume(ar_session);

Play back a different session

To play back a different dataset, pause the session and specify the new dataset before resuming the session.

ArStatus status = ArSession_pause(ar_session);
// Pause and specify a *different* dataset:
status = ArSession_setPlaybackDataset(ar_session, "/sdcard/recording2.mp4");
// Playback starts from the *beginning* of the new dataset.
status = ArSession_resume(ar_session);

Check playback status

You can use ArSession_getPlaybackStatus(ArSession*, ArPlaybackStatus*) at any time to determine the current ArPlaybackStatus.

ArPlaybackStatus playback_status;
// Can be called at any time.
ArSession_getPlaybackStatus(ar_session, &playback_status);