Recognize and Augment Images

Augmented Images in ARCore lets you build AR apps that can respond to 2D images in the user's environment, such as posters or product packaging.

You provide a set of reference images, and ARCore tells you where those images are physically located in the world.

Capabilities

The Augmented Images API has these capabilities:

  • ARCore can respond to and track images that are fixed in place, such as a print hanging on a wall or a magazine on a table. Beginning with ARCore 1.9, ARCore also tracks moving images such as an advertisement on a passing bus or an image on a flat object held by the user as they move their hands around.
  • After ARCore begins tracking an image, it provides estimates for position, orientation, and physical size. These estimates are continuously refined as ARCore gathers more data. Providing an accurate physical size improves image detection time. If no physical size is specified, ARCore estimates the size.

  • Once an image has been detected, ARCore is able to continue tracking the image's position and orientation, even though the image has temporarily moved out of the camera view.

Beginning with ARCore 1.9, you can use AugmentedImageTrackingMethod to determine whether ARCore is currently tracking the image based on where the image is currently being detected in the camera view (FullTracking), or based on its most recently detected position in the world (LastKnownPose).

  • ARCore tracks up to 20 images simultaneously in the user's environment off-camera and on-camera, but does not track multiple instances of the same image.

  • All tracking happens on the device, so no internet connection is required. Reference images can be updated on-device or over the network without requiring an app update.

  • ARCore can store feature point information for up to 1,000 reference images per image database. There's no limit to the number of databases, however only one database can be active at any given time.

  • Images can be added to an image database at runtime, up to the 1,000 per database limit.

Requirements

Images must:

  • Fill at least 25% of the camera frame to be initially detected.
  • Be flat (for example, not wrinkled or wrapped around a bottle).
  • Be in clear view of the camera. They should not be be partially obscured, viewed at an oblique angle, or viewed when the camera is moving too fast.

Best practices

Tips for selecting reference images

  • Augmented Images supports PNG and JPEG file formats. For JPEG files, avoid heavy compression for best performance.
  • Detection is solely based on points of high contrast, so both color and black & white images are detected, regardless of whether a color or black & white reference image is used.
  • The image's resolution should be at least 300 x 300 pixels.
  • Using images with high resolution does not improve performance.
  • Avoid images with sparse features.
  • Avoid images with repetitive features.
  • Use the arcoreimg tool to get a score between 0 and 100 for each image. We recommend a score of at least 75. Here are two examples:
Score: 100Score: 0
Image contains many unique feature points Image contains highly repetitive features

Tips for creating the image database

  • The database stores a compressed representation of the reference images. Each image occupies ~6KB.
  • It takes ~30ms to add an image to the database at runtime. Add images on a worker thread to avoid blocking the UI thread, or if possible, add images at compile time with the arcoreimg tool.
  • If you know the expected physical size of an image, specify it. This metadata improves tracking performance, especially for large physical images (over 75cm).
  • Avoid keeping unused images in the database as there's a slight impact on system performance.

Tips for optimizing tracking

  • Beginning with ARCore 1.9, if your image will never move from its position (for example, a poster affixed to a wall), you can attach a global anchor to the image to increase the stability of tracking.
  • The physical image must occupy 25% of the camera image. You can prompt users to fit the physical image in their camera frame with the FitToScan asset. See the Augmented Images sample app for an example of this prompt.
  • When an image is initially detected by ARCore, and no expected physical size was specified, its tracking state will be paused. This means that ARCore has recognized the image, but has not gathered enough data to estimate its location in 3D space. Do not use the image's pose and size estimates until the image's tracking state is tracking.

Send feedback about...