Use buildings and terrain around you on iOS

AI-generated Key Takeaways

• The Streetscape Geometry APIs use Google Street View data to provide 3D geometry of terrain and buildings for use in AR experiences.

• Enable the Geospatial and Streetscape Geometry modes in your ARCore session configuration to access this data.

• GARStreetscapeGeometry objects represent terrain or buildings, providing mesh data and quality information for rendering.

• You can attach AR content to Streetscape Geometry using anchors, ensuring the content is anchored to real-world structures.

• Hit-testing against Streetscape Geometry allows you to interact with and place AR objects on real-world surfaces represented by the geometry.

The Streetscape Geometry APIs provide the geometry of terrain, buildings, or other structures in a scene. The geometry can be used for occlusion, rendering, or placing AR content via hit-test APIs. Streetscape Geometry data is obtained through Google Street View imagery.

Try the sample

The GeospatialExample app demonstrates how to obtain and render Streetscape Geometries.

Set up the Geospatial API

To use Streetscape Geometry, you'll need to set up the Geospatial API in your project. Follow instructions on Enabling the Geospatial API to set up the Geospatial API.

Enable Streetscape Geometry

The Geospatial API obtains Streetscape Geometry data when the GARGeospatialMode is set to GARGeospatialModeEnabled and GARStreetscapeGeometryMode is set to GARStreetscapeGeometryModeEnabled.

GARSessionConfiguration *configuration = [[GARSessionConfiguration alloc] init];
configuration.geospatialMode = GARGeospatialModeEnabled;
configuration.streetscapeGeometryMode = GARStreetscapeGeometryModeEnabled;
[garSession setConfiguration:configuration error:&error];

Obtain Streetscape Geometry in an ARCore session

Use GARFrame.streetscapeGeometries to obtain all GARStreetscapeGeometry objects.

Understand GARStreetscapeGeometry

GARStreetscapeGeometry contains information about a building:

• GARStreetscapeGeometry.type
  Identifies the StreetscapeGeometry as either terrain or a building.
• GARStreetscapeGeometry.mesh
  Obtain a polygon GARMesh that corresponds to this terrain or building.
• GARStreetscapeGeometry.meshTransform
  Describes the origin of the geometry. All points in the GARMesh should be transformed by GARStreetscapeGeometry.meshTransform.
• GARStreetscapeGeometry.quality
  Provides the quality of the mesh data. Levels of detail are described in the CityGML 2.0 standard.

Building LOD 1

GARStreetscapeGeometryQualityBuildingLOD_1 consists of building footprints extruded upwards to a flat top. Building heights may be inaccurate.

Building LOD 2

GARStreetscapeGeometryQualityBuildingLOD_2 will have higher fidelity geometry. Mesh walls and roofs will more closely match the building's shape. Smaller features like chimneys or roof vents may still poke outside of the mesh.

Understand GARMesh

GARMesh is a polygon mesh representing a surface reconstruction of the Streetscape Geometry. Each GARMesh includes a vertex buffer and index buffer:

• GARMesh.vertexCount
  Retrieves the number of vertices in this mesh.
• GARMesh.vertices
  Obtain the concatenated positions of mesh vertices, in coordinates relative to GARStreetscapeGeometry.meshTransform.
• GARMesh.triangleCount
  Retrieves the number of indices in this mesh.
• GARMesh.triangles
  Obtain the indexes of vertices that make up a face.

Attach AR content to a GARStreetscapeGeometry

Use GARSesssion.createAnchorOnStreetscapeGeometry:transform:error: to create an anchor at a given pose near GARStreetscapeGeometry.meshTransform. This anchor will inherit its tracking state from the parent GARStreetscapeGeometry.

Perform a hit-test against GARStreetscapeGeometry

GARSession.raycastStreetscapeGeometry:direction:error: can be used to hit-test against Streetscape Geometry. If intersections are found, GARStreetscapeGeometryRaycastResult contains pose information about the hit location as well as a reference to the GARStreetscapeGeometry which was hit. This Streetscape Geometry can be passed to GARSesssion.createAnchorOnStreetscapeGeometry:transform:error: to create an anchor attached to it.

NSArray<GARStreetscapeGeometryRaycastResult *> *results =
    [session raycastStreetscapeGeometry:arRaycastQuery.origin
                              direction:arRaycastQuery.direction
                                  error:&error];
[session createAnchorOnStreetscapeGeometry:results[0].streetscapeGeometry
                                 transform:results[0].worldTransform
                                     error:&error];