AR Tracked Image Manager component

The tracked image manager is a type of trackable manager and performs 2D image tracking.

AR Tracked Image Manager component
AR Tracked Image Manager component

The tracked image manager creates GameObjects for each detected image in the environment. Before an image can be detected, the manager must be instructed to look for a set of reference images compiled into a reference image library. It only detects images in this library.

Terminology

Term Description
Reference image A reference image is an image that the XRImageTrackingSubsystem attempts to find in the real world. The subsystem associates detected images with the reference image used to detect them. Each detected image has a pose in the world.
Reference image library A set of reference images. When you start an image tracking subsystem, you must first provide it with a library of reference images so it knows what to search for.

Create a reference image library

You create reference image libraries in the Unity Editor. From Unity's main menu, go to Assets > Create > XR > Reference Image Library.

This creates a new reference image library asset in your project. To add images, select this Asset, then click Add Image and complete the fields that appear:

Editing a reference image library

You can specify the following information:

Field Description
Name A string name that is available at runtime. This name isn't used by the subsystem, but it can be useful to identify which reference image has been detected. There is no check for duplicate name conflicts.
Specify Size If enabled, you can specify the physical size you expect the image to be in the real world. This field is mandatory for some providers and optional for others. For more information, see your provider's documentation. If you specify this field, the dimensions must be greater than zero. Editing one dimension (for example, width) causes the other dimension (height) to change automatically based on the image's aspect ratio.
Keep Texture at Runtime If enabled, XRReferenceImage.texture contains a reference to the source texture. This can be useful if you need access to the source texture at runtime. By default, this is unchecked to reduce the built Player size. When unchecked, XRReferenceImage.texture is null.
Tip

You can include an XRReferenceImageLibrary in an AssetBundle and use it in an app that was not built with that reference image library present.

Reference image library

The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null. You can set it via script with:

ARTrackedImageManager manager = ...;
manager.referenceLibrary = myReferenceImageLibrary;

You can set the reference image library to be either an XRReferenceImageLibrary or a RuntimeReferenceImageLibrary. You can only create an XRReferenceImageLibrary in the Editor, and you can't modify it at runtime. A RuntimeReferenceImageLibrary is the runtime representation of an XRReferenceImageLibrary. When you set the library to be an XRReferenceImageLibrary, the image tracking subsystem automatically converts it to a RuntimeReferenceImageLibrary for consumption.

The actual image library data is provider-specific; see your provider's documentation for details.

You can create a RuntimeReferenceImageLibrary from an XRReferenceImageLibrary with the ARTrackedImageManager.CreateRuntimeLibrary method:

XRReferenceImageLibrary serializedLibrary = ...
RuntimeReferenceImageLibrary runtimeLibrary = trackedImageManager.CreateRuntimeLibrary(serializedLibrary);
Note

The ordering of the XRReferenceImages in the RuntimeReferenceImageLibrary is undefined; that is, it might not match the order in which the images appeared in the source XRReferenceImageLibrary. Each reference image does have a string name that you assign it, and a randomly assigned Guid. The Guid are the same between the source XRReferenceImageLibrary and its corresponding RuntimeReferenceImageLibrary.

Use reference image libraries with AssetBundles

To include an XRReferenceImageLibrary in an AssetBundle, AR Foundation requires that you first call ARBuildProcessor.PreprocessBuild before you build the AssetBundles. Platform-specific data is attached to XRReferenceImageLibrary assets during this preprocessor step.

Refer to the code sample below to understand how to build AssetBundles that contain reference image libraries:

static void ExportAssetBundles(string directoryPath, BuildTarget buildTarget)
{
    UnityEditor.XR.ARSubsystems.ARBuildProcessor.PreprocessBuild(buildTarget);
    BuildPipeline.BuildAssetBundles(
        directoryPath,
        BuildAssetBundleOptions.ForceRebuildAssetBundle,
        buildTarget);
}

Respond to detected images

Subscribe to the ARTrackedImageManager's trackedImagesChanged event to be notified whenever an image is added (that is, first detected), updated, or removed:

[SerializeField]
ARTrackedImageManager m_TrackedImageManager;

void OnEnable() => m_TrackedImageManager.trackedImagesChanged += OnChanged;

void OnDisable() => m_TrackedImageManager.trackedImagesChanged -= OnChanged;

void OnChanged(ARTrackedImagesChangedEventArgs eventArgs)
{
    foreach (var newImage in eventArgs.added)
    {
        // Handle added event
    }

    foreach (var updatedImage in eventArgs.updated)
    {
        // Handle updated event
    }

    foreach (var removedImage in eventArgs.removed)
    {
        // Handle removed event
    }
}

Note that images also have a tracking state which can provide additional information about the tracking quality. An image that goes out of view, for example, might not be "removed", but its tracking state likely changes.

You can also get all the currently tracked images with the ARTrackedImageManager's trackables property. This acts like an IEnumerable collection, so you can use it in a foreach statement:

void ListAllImages()
{
    Debug.Log(
        $"There are {m_TrackedImageManager.trackables.count} images being tracked.");

    foreach (var trackedImage in m_TrackedImageManager.trackables)
    {
        Debug.Log($"Image: {trackedImage.referenceImage.name} is at " +
                  $"{trackedImage.transform.position}");
    }
}

Or access a specific image by its TrackableId:

ARTrackedImage GetImageAt(TrackableId trackableId)
{
    return m_TrackedImageManager.trackables[trackableId];
}

Tracked Image Prefab

The ARTrackedImageManager has a "Tracked Image Prefab" field; however, this is not intended for content. When an image is detected, ARFoundation creates a new GameObject to represent it.

If "Tracked Image Prefab" is null, then AR Foundation creates a GameObject with an ARTrackedImage component on it. However, if you want every tracked image to also include additional components, you can provide a Prefab for AR Foundation to instantiate for each detected image. In other words, the purpose of the Prefab field is to extend the default behavior of tracked images; it is not the recommended way to place content in the world.

If you would like to instantiate content at the pose of the detected image and have its pose updated automatically, then you should parent your content to the ARTrackedImage.

Add new reference images at runtime

Some subsystems might support image libraries that are modifiable at runtime. In this case, the subsystem produces a RuntimeReferenceImageLibrary that is a MutableRuntimeReferenceImageLibrary. To use it, you need to cast the RuntimeReferenceImageLibrary to a MutableRuntimeReferenceImageLibrary:

[SerializeField]
ARTrackedImageManager m_TrackedImageManager;

void AddImage(Texture2D imageToAdd)
{
    if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
        return; // Session state is invalid

    if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        mutableLibrary.ScheduleAddImageWithValidationJob(
            imageToAdd,
            "my new image",
            0.5f /* 50 cm */);
    }
}

To create an empty library that you can add images to later, you can call CreateRuntimeLibrary without arguments:

void AddImage(Texture2D imageToAdd)
{
    if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
        return; // Session state is invalid

    var library = m_TrackedImageManager.CreateRuntimeLibrary();
    if (library is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        mutableLibrary.ScheduleAddImageWithValidationJob(
            imageToAdd,
            "my new image",
            0.5f /* 50 cm */);
    }
}

You can check whether a particular tracked image manager supports mutable libraries with its descriptor:

bool DoesSupportMutableImageLibraries()
{
    return m_TrackedImageManager.descriptor.supportsMutableLibrary;
}

You can add images to mutable libraries allow images at any time. Adding an image can be computationally resource-intensive, and might take a few frames to complete. The Unity Job System is used to process images asynchronously.

To add an image to a MutableRuntimeReferenceImageLibrary, use the ScheduleAddImageJob method. This returns a JobHandle that you can use to determine when the job is complete. You can safely discard this handle if you don't need to do this.

If you use the extension method which accepts a Texture2D, you do not need to manage any memory.

If you use the version of ScheduleAddImageJob that accepts a NativeSlice or pointer, you are responsible for managing the memory, that is, freeing it when the job completes. You can do this by scheduling a dependent job that frees the memory:

struct DeallocateJob : IJob
{
    [DeallocateOnJobCompletion]
    public NativeArray<byte> data;

    public void Execute() { }
}

void AddImage(NativeArray<byte> grayscaleImageBytes,
              int widthInPixels, int heightInPixels,
              float widthInMeters)
{
    if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
        return; // Session state is invalid

    if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        var aspectRatio = (float)widthInPixels / (float)heightInPixels;
        var sizeInMeters = new Vector2(widthInMeters, widthInMeters * aspectRatio);
        var referenceImage = new XRReferenceImage(
            // Guid is assigned after image is added
            SerializableGuid.empty,
            // No texture associated with this reference image
            SerializableGuid.empty,
            sizeInMeters, "My Image", null);

        var jobState = mutableLibrary.ScheduleAddImageWithValidationJob(
            grayscaleImageBytes,
            new Vector2Int(widthInPixels, heightInPixels),
            TextureFormat.R8,
            referenceImage);

        // Schedule a job that deallocates the image bytes after the image
        // is added to the reference image library.
        new DeallocateJob { data = grayscaleImageBytes }.Schedule(jobState.jobHandle);
    }
    else
    {
        // Cannot add the image, so dispose its memory.
        grayscaleImageBytes.Dispose();
    }
}

Multiple add image jobs can be processed concurrently. Whether or not MutableRuntimeReferenceImageLibrary is currently in use for image tracking has no effect on this.

Reference images and texture import settings

Image detection algorithms rely on accurate knowledge of a reference image's aspect ratio; however, Unity's default texture import setting adjusts texture resolutions to the nearest powers of 2 (PoT). This can have a negative effect on tracking capabilities when tracking non-power of 2 (NPoT) images.

When a NPoT image is imported with default settings and added to a MutableRuntimeReferenceImageLibrary during runtime, it may display squashing, stretching, or scaling artifacts.

To ensure images maintain their aspect ratio when imported, select the image in your project assets and change the Non-Power of 2 texture import setting to None.

Note

For textures with power-of 2 aspect ratios, Non-Power of 2 setting will not be selectable as no adjustments are necessary.

Non-Power of Two Reference Image Settings

Create a manager at runtime

When you add a component to an active GameObject at runtime, Unity immediately invokes its OnEnable method. However, the ARTrackedImageManager requires a non-null reference image library. If the reference image library is null when the ARTrackedImageManager is enabled, it automatically disables itself.

To add an ARTrackedImageManager at runtime, set its reference image library and then re-enable it:

var manager = gameObject.AddComponent<ARTrackedImageManager>();
manager.referenceLibrary = myLibrary;
manager.enabled = true;

Maximum number of moving images

Some providers can track moving images. This typically requires more CPU resources, so you can specify the number of moving images to track simultaneously. Check for support via the SubsystemDescriptor (ARTrackedImageManager.descriptor).

Tracked Image Prefab

This Prefab is instantiated whenever an image from the reference image library is detected. The manager ensures the instantiated GameObject includes an ARTrackedImage component. You can get the reference image that was used to detect the ARTrackedImage with the ARTrackedImage.referenceImage property.

Tracking State

There are three possible tracking states for ARTrackedImages:

TrackingState Description
None The image is not being tracked. Note that this may be the initial state when the image is first detected.
Limited The image is being tracked, but not as effectively. The situations in which an image is considered Limited instead of Tracking depend on the underlying AR framework. Examples that could cause Limited tracking include:
  • Obscuring the image so that it is not visible to the camera.
  • The image is not tracked as a moving image. This can happen, for example, if the maxNumberOfMovingImages is exceeded.
Tracking The underlying AR SDK reports that it is actively tracking the image.

Determining when an image is visible

There is no API to determine the visibility of an image. Generally, if the tracking state is Tracking, it likely changes to Limited when the image is not visible. However, there are other situations in which the tracking state can be in states other than Tracked.

If this information is important to your application, considering comparing the ARTrackedImage's transform with the camera's view frustum.