Google is committed to advancing racial equity for Black communities. See how.

ArFrame

Per-frame state.

Summary

Enumerations

ArCoordinates2dType{
  AR_COORDINATES_2D_TEXTURE_TEXELS = 0,
  AR_COORDINATES_2D_TEXTURE_NORMALIZED = 1,
  AR_COORDINATES_2D_IMAGE_PIXELS = 2,
  AR_COORDINATES_2D_IMAGE_NORMALIZED = 3,
  AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES = 6,
  AR_COORDINATES_2D_VIEW = 7,
  AR_COORDINATES_2D_VIEW_NORMALIZED = 8
}
enum
2d coordinate systems supported by ARCore.

Typedefs

ArFrame typedef
struct ArFrame_
The world state resulting from an update (value type).

Functions

ArFrame_acquireCamera(const ArSession *session, const ArFrame *frame, ArCamera **out_camera)
void
Returns the camera object for the session.
ArFrame_acquireCameraImage(ArSession *session, ArFrame *frame, ArImage **out_image)
Returns the CPU image for the current frame.
ArFrame_acquireDepthImage(const ArSession *session, const ArFrame *frame, ArImage **out_depth_image)
Attempts to acquire a depth image that corresponds to the current frame.
ArFrame_acquireImageMetadata(const ArSession *session, const ArFrame *frame, ArImageMetadata **out_metadata)
Gets the camera metadata for the current camera image.
ArFrame_acquirePointCloud(const ArSession *session, const ArFrame *frame, ArPointCloud **out_point_cloud)
Acquires the current set of estimated 3d points attached to real-world geometry.
ArFrame_create(const ArSession *session, ArFrame **out_frame)
void
Allocates a new ArFrame object, storing the pointer into *out_frame.
ArFrame_destroy(ArFrame *frame)
void
Releases an ArFrame and any references it holds.
ArFrame_getAndroidSensorPose(const ArSession *session, const ArFrame *frame, ArPose *out_pose)
void
Sets out_pose to the pose of the Android Sensor Coordinate System in the world coordinate space for this frame.
ArFrame_getCameraTextureName(const ArSession *session, const ArFrame *frame, uint32_t *out_texture_id)
void
Returns the OpenGL ES camera texture name (ID) associated with this frame.
ArFrame_getDisplayGeometryChanged(const ArSession *session, const ArFrame *frame, int32_t *out_geometry_changed)
void
Checks if the display rotation or viewport geometry changed since the previous call to ArSession_update.
ArFrame_getLightEstimate(const ArSession *session, const ArFrame *frame, ArLightEstimate *out_light_estimate)
void
Gets the current ArLightEstimate, if Lighting Estimation is enabled.
ArFrame_getTimestamp(const ArSession *session, const ArFrame *frame, int64_t *out_timestamp_ns)
void
Returns the timestamp in nanoseconds when this image was captured.
ArFrame_getUpdatedAnchors(const ArSession *session, const ArFrame *frame, ArAnchorList *out_anchor_list)
void
Gets the set of anchors that were changed by the ArSession_update that produced this Frame.
ArFrame_getUpdatedTrackables(const ArSession *session, const ArFrame *frame, ArTrackableType filter_type, ArTrackableList *out_trackable_list)
void
Gets the set of trackables of a particular type that were changed by the ArSession_update call that produced this Frame.
ArFrame_hitTest(const ArSession *session, const ArFrame *frame, float pixel_x, float pixel_y, ArHitResultList *hit_result_list)
void
Performs a ray cast from the user's device in the direction of the given location in the camera view.
ArFrame_hitTestInstantPlacement(const ArSession *session, const ArFrame *frame, float pixel_x, float pixel_y, float approximate_distance_meters, ArHitResultList *hit_result_list)
void
Performs a ray cast that can return a result before ARCore establishes full tracking.
ArFrame_hitTestRay(const ArSession *session, const ArFrame *frame, const float *ray_origin_3, const float *ray_direction_3, ArHitResultList *hit_result_list)
void
Similar to ArFrame_hitTest, but takes an arbitrary ray in world space coordinates instead of a screen space point.
ArFrame_transformCoordinates2d(const ArSession *session, const ArFrame *frame, ArCoordinates2dType input_coordinates, int32_t number_of_vertices, const float *vertices_2d, ArCoordinates2dType output_coordinates, float *out_vertices_2d)
void
Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.
ArFrame_transformDisplayUvCoords(const ArSession *session, const ArFrame *frame, int32_t num_elements, const float *uvs_in, float *uvs_out)
void
Deprecated. Deprecated in release 1.7.0. Use ArFrame_transformCoordinates2d instead.
Transform the given texture coordinates to correctly show the background image.

Enumerations

ArCoordinates2dType

 ArCoordinates2dType

2d coordinate systems supported by ARCore.

Properties
AR_COORDINATES_2D_IMAGE_NORMALIZED

CPU image, (x,y) normalized to [0.0f, 1.0f] range.

AR_COORDINATES_2D_IMAGE_PIXELS

CPU image, (x,y) in pixels.

The range of x and y is determined by the CPU image resolution.

AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES

OpenGL Normalized Device Coordinates, display-rotated, (x,y) normalized to [-1.0f, 1.0f] range.

AR_COORDINATES_2D_TEXTURE_NORMALIZED

GPU texture coordinates, (s,t) normalized to [0.0f, 1.0f] range.

AR_COORDINATES_2D_TEXTURE_TEXELS

GPU texture, (x,y) in pixels.

AR_COORDINATES_2D_VIEW

Android view, display-rotated, (x,y) in pixels.

AR_COORDINATES_2D_VIEW_NORMALIZED

Android view, display-rotated, (x,y) normalized to [0.0f, 1.0f] range.

Typedefs

ArFrame

struct ArFrame_ ArFrame

The world state resulting from an update (value type).

Functions

ArFrame_acquireCamera

void ArFrame_acquireCamera(
  const ArSession *session,
  const ArFrame *frame,
  ArCamera **out_camera
)

Returns the camera object for the session.

Note that this Camera instance is long-lived so the same instance is returned regardless of the frame object this function was called on.

ArFrame_acquireCameraImage

ArStatus ArFrame_acquireCameraImage(
  ArSession *session,
  ArFrame *frame,
  ArImage **out_image
)

Returns the CPU image for the current frame.

Caller is responsible for later releasing the image with ArImage_release. Not supported on all devices (see https://developers.google.com/ar/discover/supported-devices). Return values:

Details
Returns
AR_SUCCESS or any of:

ArFrame_acquireDepthImage

ArStatus ArFrame_acquireDepthImage(
  const ArSession *session,
  const ArFrame *frame,
  ArImage **out_depth_image
)

Attempts to acquire a depth image that corresponds to the current frame.

The depth image has a single 16-bit plane at index 0. Each pixel contains the distance in millimeters to the camera plane. Currently, only the low order 13 bits are used. The 3 highest order bits are always set to 000. The image plane is stored in big-endian format. The actual resolution of the depth image depends on the device and its display aspect ratio, with sizes typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future.

The output depth image can express depth values from 0 millimeters to 8191 millimeters. Optimal depth accuracy is achieved between 50 millimeters and 5000 millimeters from the camera. Error increases quadratically as distance from the camera increases.

Depth is estimated using data from previous frames and the current frame. As the user moves their device through the environment 3D depth data is collected and cached, improving the quality of subsequent depth images and reducing the error introduced by camera distance.

If an up to date depth image isn't ready for the current frame, the most recent depth image available from an earlier frame is returned instead. This is only expected to occur on compute-constrained devices. An up to date depth image should typically become available again within a few frames. Compare ArImage_getTimestamp depth image timestamp with the ArFrame_getTimestamp frame timestamp to determine which camera frame the depth image corresponds to.

The image must be released with ArImage_release once it is no longer needed.

Details
Parameters
session
The ARCore session.
frame
The current frame.
out_depth_image
On successful return, this is filled out with a pointer to an ArImage. On error return, this is filled out with nullptr.
Returns
AR_SUCCESS or any of:

ArFrame_acquireImageMetadata

ArStatus ArFrame_acquireImageMetadata(
  const ArSession *session,
  const ArFrame *frame,
  ArImageMetadata **out_metadata
)

Gets the camera metadata for the current camera image.

Details
Returns
AR_SUCCESS or any of:

ArFrame_acquirePointCloud

ArStatus ArFrame_acquirePointCloud(
  const ArSession *session,
  const ArFrame *frame,
  ArPointCloud **out_point_cloud
)

Acquires the current set of estimated 3d points attached to real-world geometry.

A matching call to ArPointCloud_release must be made when the application is done accessing the Point Cloud.

Note: This information is for visualization and debugging purposes only. Its characteristics and format are subject to change in subsequent versions of the API.

Details
Parameters
session
The ARCore session.
frame
The current frame.
out_point_cloud
Pointer to an ArPointCloud* receive the address of the Point Cloud.
Returns
AR_SUCCESS or any of:

ArFrame_create

void ArFrame_create(
  const ArSession *session,
  ArFrame **out_frame
)

Allocates a new ArFrame object, storing the pointer into *out_frame.

Note: the same ArFrame can be used repeatedly when calling ArSession_update.

ArFrame_destroy

void ArFrame_destroy(
  ArFrame *frame
)

Releases an ArFrame and any references it holds.

ArFrame_getAndroidSensorPose

void ArFrame_getAndroidSensorPose(
  const ArSession *session,
  const ArFrame *frame,
  ArPose *out_pose
)

Sets out_pose to the pose of the Android Sensor Coordinate System in the world coordinate space for this frame.

The orientation follows the device's "native" orientation (it is not affected by display rotation) with all axes corresponding to those of the Android sensor coordinates.

See Also:

Note: This pose is only useful when ArCamera_getTrackingState returns AR_TRACKING_STATE_TRACKING and otherwise should not be used.

Details
Parameters
session
The ARCore session
frame
The current frame.
out_pose
An already-allocated ArPose object into which the pose will be stored.

ArFrame_getCameraTextureName

void ArFrame_getCameraTextureName(
  const ArSession *session,
  const ArFrame *frame,
  uint32_t *out_texture_id
)

Returns the OpenGL ES camera texture name (ID) associated with this frame.

This is guaranteed to be one of the texture names previously set via ArSession_setCameraTextureNames or ArSession_setCameraTextureName. Texture names (IDs) are returned in a round robin fashion in sequential frames.

Details
Parameters
session
The ARCore session.
frame
The current frame.
out_texture_id
Where to store the texture name (ID).

ArFrame_getDisplayGeometryChanged

void ArFrame_getDisplayGeometryChanged(
  const ArSession *session,
  const ArFrame *frame,
  int32_t *out_geometry_changed
)

Checks if the display rotation or viewport geometry changed since the previous call to ArSession_update.

The application should re-query ArCamera_getProjectionMatrix and ArFrame_transformCoordinates2d whenever this emits non-zero.

ArFrame_getLightEstimate

void ArFrame_getLightEstimate(
  const ArSession *session,
  const ArFrame *frame,
  ArLightEstimate *out_light_estimate
)

Gets the current ArLightEstimate, if Lighting Estimation is enabled.

Details
Parameters
session
The ARCore session.
frame
The current frame.
out_light_estimate
The ArLightEstimate to fill. This object must have been previously created with ArLightEstimate_create.

ArFrame_getTimestamp

void ArFrame_getTimestamp(
  const ArSession *session,
  const ArFrame *frame,
  int64_t *out_timestamp_ns
)

Returns the timestamp in nanoseconds when this image was captured.

This can be used to detect dropped frames or measure the camera frame rate. The time base of this value is specifically not defined, but it is likely similar to clock_gettime(CLOCK_BOOTTIME).

ArFrame_getUpdatedAnchors

void ArFrame_getUpdatedAnchors(
  const ArSession *session,
  const ArFrame *frame,
  ArAnchorList *out_anchor_list
)

Gets the set of anchors that were changed by the ArSession_update that produced this Frame.

Details
Parameters
session
The ARCore session
frame
The current frame.
out_anchor_list
The list to fill. This list must have already been allocated with ArAnchorList_create. If previously used, the list is cleared first.

ArFrame_getUpdatedTrackables

void ArFrame_getUpdatedTrackables(
  const ArSession *session,
  const ArFrame *frame,
  ArTrackableType filter_type,
  ArTrackableList *out_trackable_list
)

Gets the set of trackables of a particular type that were changed by the ArSession_update call that produced this Frame.

Details
Parameters
session
The ARCore session
frame
The current frame.
filter_type
The type(s) of trackables to return. See ArTrackableType for legal values.
out_trackable_list
The list to fill. This list must have already been allocated with ArTrackableList_create. If previously used, the list is cleared first.

ArFrame_hitTest

void ArFrame_hitTest(
  const ArSession *session,
  const ArFrame *frame,
  float pixel_x,
  float pixel_y,
  ArHitResultList *hit_result_list
)

Performs a ray cast from the user's device in the direction of the given location in the camera view.

Intersections with detected scene geometry are returned, sorted by distance from the device; the nearest intersection is returned first.

Note: Significant geometric leeway is given when returning hit results. For example, a plane hit may be generated if the ray came close, but did not actually hit within the plane extents or plane bounds (ArPlane_isPoseInExtents and ArPlane_isPoseInPolygon can be used to determine these cases). A point (Point Cloud) hit is generated when a point is roughly within one finger-width of the provided screen coordinates.

The resulting list is ordered by distance, with the nearest hit first

Note: If not tracking, the hit_result_list will be empty.

Note: If called on an old frame (not the latest produced by ArSession_update the hit_result_list will be empty).

Note: When using AR_SESSION_FEATURE_FRONT_CAMERA, the returned hit result list will always be empty, as the camera is not AR_TRACKING_STATE_TRACKING. Hit testing against tracked faces is not currently supported.

Details
Parameters
session
The ARCore session.
frame
The current frame.
pixel_x
Logical X position within the view, as from an Android UI event.
pixel_y
Logical Y position within the view, as from an Android UI event.
hit_result_list
The list to fill. This list must have been previously allocated using ArHitResultList_create. If the list has been previously used, it will first be cleared.

ArFrame_hitTestInstantPlacement

void ArFrame_hitTestInstantPlacement(
  const ArSession *session,
  const ArFrame *frame,
  float pixel_x,
  float pixel_y,
  float approximate_distance_meters,
  ArHitResultList *hit_result_list
)

Performs a ray cast that can return a result before ARCore establishes full tracking.

The pose and apparent scale of attached objects depends on the ArInstantPlacementPoint tracking method and the provided approximate_distance_meters. A discussion of the different tracking methods and the effects of apparent object scale are described in ArInstantPlacementPoint.

This function will succeed only if ArInstantPlacementMode is AR_INSTANT_PLACEMENT_MODE_LOCAL_Y_UP in the ARCore session configuration, the ARCore session tracking state is AR_TRACKING_STATE_TRACKING, and there are sufficient feature points to track the point in screen space.

Details
Parameters
session
The ARCore session.
frame
The current frame.
pixel_x
Logical X position within the view, as from an Android UI event.
pixel_y
Logical Y position within the view, as from an Android UI event.
approximate_distance_meters
The distance at which to create an ArInstantPlacementPoint. This is only used while the tracking method for the returned point is AR_INSTANT_PLACEMENT_POINT_TRACKING_METHOD_SCREENSPACE_WITH_APPROXIMATE_DISTANCE.
hit_result_list
The list to fill. If successful the list will contain a single ArHitResult, otherwise it will be cleared. The ArHitResult will have a trackable of type ArInstantPlacementPoint. The list must have been previously allocated using ArHitResultList_create.

ArFrame_hitTestRay

void ArFrame_hitTestRay(
  const ArSession *session,
  const ArFrame *frame,
  const float *ray_origin_3,
  const float *ray_direction_3,
  ArHitResultList *hit_result_list
)

Similar to ArFrame_hitTest, but takes an arbitrary ray in world space coordinates instead of a screen space point.

Details
Parameters
session
The ARCore session.
frame
The current frame.
ray_origin_3
A pointer to float[3] array containing ray origin in world space coordinates.
ray_direction_3
A pointer to float[3] array containing ray direction in world space coordinates. Does not have to be normalized.
hit_result_list
The list to fill. This list must have been previously allocated using ArHitResultList_create. If the list has been previously used, it will first be cleared.

ArFrame_transformCoordinates2d

void ArFrame_transformCoordinates2d(
  const ArSession *session,
  const ArFrame *frame,
  ArCoordinates2dType input_coordinates,
  int32_t number_of_vertices,
  const float *vertices_2d,
  ArCoordinates2dType output_coordinates,
  float *out_vertices_2d
)

Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.

For Android view coordinates (AR_COORDINATES_2D_VIEW, AR_COORDINATES_2D_VIEW_NORMALIZED), the view information is taken from the most recent call to ArSession_setDisplayGeometry.

Must be called on the most recently obtained ArFrame object. If this function is called on an older frame, a log message will be printed and out_vertices_2d will remain unchanged.

Some examples of useful conversions:

If inputCoordinates is same as outputCoordinates, the input vertices will be copied to the output vertices unmodified.

Details
Parameters
session
The ARCore session.
frame
The current frame.
input_coordinates
The coordinate system used by vectors2d_in.
number_of_vertices
The number of 2D vertices to transform. vertices_2d and out_vertices_2d must point to arrays of size at least number_of_vertices * 2.
vertices_2d
Input 2D vertices to transform.
output_coordinates
The coordinate system to convert to.
out_vertices_2d
Transformed 2d vertices, can be the same array as vertices_2d for in-place transform.

ArFrame_transformDisplayUvCoords

void ArFrame_transformDisplayUvCoords(
  const ArSession *session,
  const ArFrame *frame,
  int32_t num_elements,
  const float *uvs_in,
  float *uvs_out
)

Transform the given texture coordinates to correctly show the background image.

This accounts for the display rotation, and any additional required adjustment. For performance, this function should be called only if ArFrame_getDisplayGeometryChanged indicates a change.

Deprecated. Deprecated in release 1.7.0. Use ArFrame_transformCoordinates2d instead.

Details
Parameters
session
The ARCore session
frame
The current frame.
num_elements
The number of floats to transform. Must be a multiple of 2. uvs_in and uvs_out must point to arrays of at least this many floats.
uvs_in
Input UV coordinates in normalized screen space.
uvs_out
Output UV coordinates in texture coordinates.