Unity How-to Guide: Area Learning

In this Area Learning How-to guide, you'll learn how to use a saved area description, create a new one, and extend existing ones. If you're not already familiar with Area Learning, see the introduction.

This guide makes extensive use of prefabs. If you want to learn more about them, see the Unity documentation.

Part 1: Using a saved area description

In this section, you'll learn how to use previously created area descriptions in your application. Specifically, you'll be able to:

  • Get a list of area descriptions.
  • Load a specific area description.
  • Show the user an instruction related to relocalization.

If you don't have an area description on your device, or the most recent area description isn't of the area you're currently in, you'll need to create one. You can download the Tango Unity SDK, import the unitypackage file, and then build the example app "AreaLearning" located in the Assets > TangoSDK > Examples folder, or use any other application that can save area descriptions. The ability to save area descriptions in your own application is discussed later in this How-to guide.

Preliminary steps

  1. If you haven't set up your computer to develop with Tango in Unity yet, see Getting Started With Unity.

  2. This guide builds on the Unity How-to guide for motion tracking. Go to that page and complete the steps found there, stopping just before the "Build and Run" section. When you have finished, return to this page and continue on with the steps below.

Tango Manager settings for saved area descriptions

  1. In the Hierarchy panel, click Tango Manager.
  2. In the Inspector panel, deselect Auto-connect to Service.

    When this option is selected, Tango Manager handles startup and initialization automatically. This won't work for Area Learning because it requires some custom settings. Keep this option turned off in Tango Manager; later in this How-to guide, you'll learn how to make your own "Area Learning Startup" to handle these settings.

  3. Select Enable Area Descriptions.

    This requests the Area Learning permission, which is required to load an area description or get the list of area descriptions on a device. After you select Enable Area Descriptions, a secondary option appears called Mode, which is set to Load Existing. Leave this as the active setting.

Create a startup gameObject

  1. On the GameObject menu, click Create Empty.

  2. In the Inspector panel, change the name to "Startup."

Add a script to the startup gameObject

  1. In the Inspector panel, click Add Component.

  2. In the Component menu, click New Script. (You may have to scroll down to see it.)

  3. In the New Script dialog, change the name to "AreaLearningStartup".

  4. Click Create and Add.

Add the code for the AreaLearningStartup script

  1. In the Area Learning Startup (Script) component, click the gear icon to view the context menu, and then select Edit Script.

  2. In the script editor, replace the existing script with the code below. To copy the code to the Clipboard, move the pointer over the code, and then in the upper right corner of the code section, click the Click to copy button.

    using System.Collections;
    using UnityEngine;
    using Tango;
    
    public class AreaLearningStartup : MonoBehaviour, ITangoLifecycle
    {
        private TangoApplication m_tangoApplication;
    
        public void Start()
        {
            m_tangoApplication = FindObjectOfType<TangoApplication>();
            if (m_tangoApplication != null)
            {
                m_tangoApplication.Register(this);
                m_tangoApplication.RequestPermissions();
            }
        }
    
        public void OnTangoPermissions(bool permissionsGranted)
        {
            if (permissionsGranted)
            {
                AreaDescription[] list = AreaDescription.GetList();
                AreaDescription mostRecent = null;
                AreaDescription.Metadata mostRecentMetadata = null;
                if (list.Length > 0)
                {
                    // Find and load the most recent Area Description
                    mostRecent = list[0];
                    mostRecentMetadata = mostRecent.GetMetadata();
                    foreach (AreaDescription areaDescription in list)
                    {
                        AreaDescription.Metadata metadata = areaDescription.GetMetadata();
                        if (metadata.m_dateTime > mostRecentMetadata.m_dateTime)
                        {
                            mostRecent = areaDescription;
                            mostRecentMetadata = metadata;
                        }
                    }
    
                    m_tangoApplication.Startup(mostRecent);
                }
                else
                {
                    // No Area Descriptions available.
                    Debug.Log("No area descriptions available.");
                }
            }
        }
    
        public void OnTangoServiceConnected()
        {
        }
    
        public void OnTangoServiceDisconnected()
        {
        }
    }
    

Code details

As described earlier, you must manually control the startup process of the TangoApplication component on the Tango Manager gameObject.

In the AreaLearningStartup class, you declare a TangoApplication object:

private TangoApplication m_tangoApplication;

In the Start() method, you initialize m_tangoApplication to refer to the TangoApplication in the scene:

public void Start()
{
    m_tangoApplication = FindObjectOfType<TangoApplication>();

To load an area description file, you must specify it when connecting to Tango. A common way to do this is by showing a list of all area descriptions on the device for the user to choose from. However, you can only query the list of area descriptions after you are granted the Area Learning permission. Therefore, you must manually control the startup process by explicitly calling the RequestPermissions() and Startup() methods on the TangoApplication component on the Tango Manager gameObject.

Register to receive callbacks, and request permissions:

if (m_tangoApplication != null)
{
    m_tangoApplication.Register(this);
    m_tangoApplication.RequestPermissions();
}

After permissions are granted, OnTangoPermissions() is called and you can get the list of area descriptions and then connect to a specified area description. For simplicity, this example uses the most recent area description:

if (permissionsGranted)
{
    AreaDescription[] list = AreaDescription.GetList();
    AreaDescription mostRecent = null;
    AreaDescription.Metadata mostRecentMetadata = null;
    if (list.Length > 0)
    {
        // Find and load the most recent Area Description
        mostRecent = list[0];
        mostRecentMetadata = mostRecent.GetMetadata();
        foreach (AreaDescription areaDescription in list)
        {
            AreaDescription.Metadata metadata = areaDescription.GetMetadata();
            if (metadata.m_dateTime > mostRecentMetadata.m_dateTime)
            {
                mostRecent = areaDescription;
                mostRecentMetadata = metadata;
            }
        }

You connect to Tango and pass the most recent area description:

m_tangoApplication.Startup(mostRecent);

For your application, you will need to implement more functionality in the OnTangoPermissions function. In this example, we always pick the most recent area description and only write a log if there are no visible area descriptions. For your application, you should implement your own logic for choosing an area description and handling the case where there is no area description available.

When you have finished examining the script, save it, and then return to the Unity Editor.

Create an instruction screen

Before localization happens, the application gets no motion updates. To improve the user experience, you can instruct the user to walk around until they have localized.

  1. In the Hierarchy panel, select Create > UI > Canvas.

  2. In the Hierarchy panel, right-click Canvas and then select UI > Image.

  3. In the Project panel, navigate to the Assets > TangoSDK > Examples > Common > Textures folder. It should contain a texture called relocalize_screen. You can also download this texture from our Unity examples.

  4. In the Hierarchy panel, click Image.

  5. In the Image (Script) component of the Image gameObject, there is a field named Source Image. Drag the relocalize_screen texture to the Source Image field.

  6. To ensure the texture displays correctly, click Set Native Size.

Listen for localization and show the instruction

  1. In the Project panel, navigate to the Assets > TangoSDK > Examples > AreaLearning > Scripts folder. It should contain a script called RelocalizingOverlay.

  2. In the Hierarchy panel, click Tango Manager.

  3. Drag the RelocalizingOverlay script to a blank space in the Tango Manager's Inspector panel.

  4. In the Relocalizing Overlay (Script) component you just created, there is a field named Relocalization Overlay. Drag the Image gameObject from the Hierarchy panel to the Relocalization Overlay field.

Build and run your project

  1. On the File menu, click Build and Run.

  2. In the Build Settings dialog, click Build and Run.

  3. In the Build Android dialog, enter a name for your app in the Save As field, and then click Save.

Your application loads the most recent area description and shows a screen instructing you to walk around until relocalization occurs. After that happens, the instruction screen disappears and you can walk around and view the plane, cube, and sphere where you placed them in the scene. The app uses the area description's coordinate frame while handling camera movement.

Part 2: Create and extend area descriptions

In this section, you'll learn how to use Area Learning mode to create new area descriptions or extend existing ones. It builds on the steps you've followed thus far in this How-to guide.

Tango Manager settings for new area descriptions

  1. In the Hierarchy panel, click Tango Manager.

  2. In the Inspector panel, beneath Enable Area Descriptions, click the field to the right of Mode, and then click Learning.

    Keep in mind that in learning mode, your app selects key features in each incoming image and memorizes them to give you the benefits of drift correction and localization. This requires much more computational power than loading an area description file. You should restrict Learning mode to a separate setup mode in your application and use Load Existing mode during your main experience.

    In Learning mode, you can use the same startup script as Load Existing mode with one key difference in the behavior of m_tangoApplication.Startup(). In Learning mode, calling m_tangoApplication.Startup(null) is valid, and triggers the creation of a new area description.

  3. In the AreaLearningStartup script (attached to the Startup gameObject), replace the line:

    Debug.Log("No area descriptions available.");
    

    with:

    m_tangoApplication.Startup(null);
    

    Your application will load the most recent area description if present, or create a new area description if none exist on the device.

Save an area description

In Learning mode, you can save the area description that was learned by calling AreaDescription.SaveCurrent() in a place that makes sense for your own application. Saving an area description can take a few minutes, so be sure to save in a background thread.

While the save is running, TangoEvent notification callbacks are sent that describe the save progress. You can listen for these callbacks by implementing the ITangoEvent interface and listening to AreaDescriptionSaveProgress events. In the snippet below, we update a GUIText m_savingText with the save progress as a percentage. You'll need to create your own implementation to match your UI.

public void OnTangoEventAvailableEventHandler(Tango.TangoEvent tangoEvent)
{
    if (tangoEvent.type == TangoEnums.TangoEventType.TANGO_EVENT_AREA_LEARNING
        && tangoEvent.event_key == "AreaDescriptionSaveProgress")
    {
        m_savingText.text = "Saving. " + (float.Parse(tangoEvent.event_value) * 100) + "%";
    }
}

You can also see saving progress implemented in our Area Learning sample.

Relocalizing

When you relocalize, small errors that have been accumulated are corrected, including errors in the past. You can use this to correct not only the device's location, but also objects that were placed relative to the device, even back in time.

The augmented reality sample does this. The sample lets you drop virtual markers into the real world. Every time a marker is dropped, the sample also stores the timestamp when you placed the marker and the transformation from the device to the placed marker. When you relocalize, the sample goes through the list of objects, asks for the (now corrected) device pose for each timestamp, and updates the object's location using the new device pose and stored transformation.

The code snippet below from our augmented reality sample shows one implementation. Note that it relies on the ARMarker class's member variables to keep the timestamp and pose.

public void OnTangoPoseAvailable(Tango.TangoPoseData poseData)
{
    if (poseData.framePair.baseFrame ==
        TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_AREA_DESCRIPTION &&
        poseData.framePair.targetFrame ==
        TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_START_OF_SERVICE &&
        poseData.status_code == TangoEnums.TangoPoseStatusType.TANGO_POSE_VALID)
    {
        // Adjust mark's position each time a loop closure is detected.
        foreach (GameObject obj in m_markerList)
        {
            ARMarker tempMarker = obj.GetComponent<ARMarker>();
            if (tempMarker.m_timestamp != -1.0f)
            {
                TangoCoordinateFramePair pair;
                TangoPoseData relocalizedPose = new TangoPoseData();

                pair.baseFrame = TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_AREA_DESCRIPTION;
                pair.targetFrame = TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_DEVICE;
                PoseProvider.GetPoseAtTime(relocalizedPose, tempMarker.m_timestamp, pair);
                Vector3 pDevice = new Vector3((float)relocalizedPose.translation[0],
                                              (float)relocalizedPose.translation[1],
                                              (float)relocalizedPose.translation[2]);
                Quaternion qDevice = new Quaternion((float)relocalizedPose.orientation[0],
                                                    (float)relocalizedPose.orientation[1],
                                                    (float)relocalizedPose.orientation[2],
                                                    (float)relocalizedPose.orientation[3]);

                Matrix4x4 uwTDevice = m_uwTss * Matrix4x4.TRS(pDevice, qDevice, Vector3.one) * m_dTuc;
                Matrix4x4 uwTMarker = uwTDevice * tempMarker.m_deviceTMarker;

                obj.transform.position = uwTMarker.GetColumn(3);
                obj.transform.rotation = Quaternion.LookRotation(uwTMarker.GetColumn(2), uwTMarker.GetColumn(1));
            }
        }
    }
}

Enviar comentarios sobre…