AR tracked image manager

The tracked image manager is a type of trackable manager and performs 2D image tracking.

The tracked image manager creates GameObjects for each detected image in the environment. Before an image can be detected, the manager must be instructed to look for a set of reference images compiled into a reference image library. It only detects images in this library.

Reference library

For instructions on how to create a reference image library in the Unity Editor, see documentation on the Tracked Image Subsystem.

The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null. You can set it via script with:

ARTrackedImageManager manager = ...;
manager.referenceLibrary = myReferenceImageLibrary;

You can set the reference image library to be either an XRReferenceImageLibrary or a RuntimeReferenceImageLibrary. You can only create an XRReferenceImageLibrary in the Editor, and you can't modify it at runtime. A RuntimeReferenceImageLibrary is the runtime representation of an XRReferenceImageLibrary. When you set the library to be an XRReferenceImageLibrary, the image tracking subsystem automatically converts it to a RuntimeReferenceImageLibrary for consumption.

The actual image library data is provider-specific; refer to your provider's documentation for details.

You can create a RuntimeReferenceImageLibrary from an XRReferenceImageLibrary with the ARTrackedImageManager.CreateRuntimeLibrary method:

XRReferenceImageLibrary serializedLibrary = ...
RuntimeReferenceImageLibrary runtimeLibrary = trackedImageManager.CreateRuntimeLibrary(serializedLibrary);

⚠️ Note The ordering of the XRReferenceImages in the RuntimeReferenceImageLibrary is undefined; that is, it may not match the order in which the images appeared in the source XRReferenceImageLibrary. Each reference image does have a string name that you assign it, and a randomly assigned Guid. The Guid are the same between the source XRReferenceImageLibrary and its corresponding RuntimeReferenceImageLibrary.

Responding to detected images

Subscribe to the ARTrackedImageManager's trackedImagesChanged event to be notified whenever an image is added (i.e., first detected), updated, or removed:

[SerializeField]
ARTrackedImageManager m_TrackedImageManager;

void OnEnable() => m_TrackedImageManager.trackedImagesChanged += OnChanged;

void OnDisable() => m_TrackedImageManager.trackedImagesChanged -= OnChanged;

void OnChanged(ARTrackedImagesChangedEventArgs eventArgs)
{
    foreach (var newImage in eventArgs.added)
    {
        // Handle added event
    }

    foreach (var updatedImage in eventArgs.updated)
    {
        // Handle updated event
    }

    foreach (var removedImage in eventArgs.removed)
    {
        // Handle removed event
    }
}

Note that images also have a tracking state which can provide additional information about the tracking quality. An image that goes out of view, for example, may not be "removed", but its tracking state will likely change.

You can also get all the currently tracked images with the ARTrackedImageManager's trackables property. This acts like an IEnumerable collection, so you can use it in a foreach statement:

void ListAllImages()
{
    Debug.Log(
        $"There are {m_TrackedImageManager.trackables.count} images being tracked.");

    foreach (var trackedImage in m_TrackedImageManager.trackables)
    {
        Debug.Log($"Image: {trackedImage.referenceImage.name} is at " +
                  $"{trackedImage.transform.position}");
    }
}

Or access a specific image by its TrackableId:

ARTrackedImage GetImageAt(TrackableId trackableId)
{
    return m_TrackedImageManager.trackables[trackableId];
}

Tracked Image Prefab

The ARTrackedImageManager has a "Tracked Image Prefab" field; however, this is not intended for content. When an image is detected, ARFoundation will create a new GameObject to represent it.

If "Tracked Image Prefab" is null, then ARFoundation simply creates a GameObject with an ARTrackedImage component on it. However, if you want every tracked image to also include additional components, you can provide a prefab for ARFoundation to instantiate for each detected image. In other words, the purpose of the prefab field is to extend the default behavior of tracked images; it is not the recommended way to place content in the world.

If you would like to instantiate content at the pose of the detected image and have its pose updated automatically, then you should parent your content to the ARTrackedImage.

Adding new reference images at runtime

Some subsystems might support image libraries that are modifiable at runtime. In this case, the subsystem produces a RuntimeReferenceImageLibrary that is a MutableRuntimeReferenceImageLibrary. To use it, you need to cast the RuntimeReferenceImageLibrary to a MutableRuntimeReferenceImageLibrary:

[SerializeField]
ARTrackedImageManager m_TrackedImageManager;

void AddImage(Texture2D imageToAdd)
{
    if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        mutableLibrary.ScheduleAddImageJob(
            imageToAdd,
            "my new image",
            0.5f /* 50 cm */);
    }
}

To create an empty library that you can add images to later, you can call CreateRuntimeLibrary without arguments:

void AddImage(Texture2D imageToAdd)
{
    var library = m_TrackedImageManager.CreateRuntimeLibrary();
    if (library is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        mutableLibrary.ScheduleAddImageJob(
            imageToAdd,
            "my new image",
            0.5f /* 50 cm */);
    }
}

You can check whether a particular tracked image manager supports mutable libraries with its descriptor:

bool DoesSupportMutableImageLibraries()
{
    return m_TrackedImageManager.descriptor.supportsMutableLibrary;
}

You can add images to mutable libraries allow images at any time. Adding an image can be computationally expensive, and might take a few frames to complete. The Unity Job System is used to process images asynchronously.

To add an image to a MutableRuntimeReferenceImageLibrary, use the ScheduleAddImageJob method. This returns a JobHandle that you can use to determine when the job is complete. You can safely discard this handle if you don't need to do this.

If you use the extension method which accepts a Texture2D, you do not need to manage any memory.

If you use the version of ScheduleAddImageJob that accepts a NativeSlice or pointer, you are responsible for managing the memory, i.e., freeing it when the job completes. You can do this by scheduling a dependent job that frees the memory:

struct DeallocateJob : IJob
{
    [DeallocateOnJobCompletion]
    public NativeArray<byte> data;

    public void Execute() { }
}

void AddImage(NativeArray<byte> grayscaleImageBytes,
              int widthInPixels, int heightInPixels,
              float widthInMeters)
{
    if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        var aspectRatio = (float)widthInPixels / (float)heightInPixels;
        var sizeInMeters = new Vector2(widthInMeters, widthInMeters * aspectRatio);
        var referenceImage = new XRReferenceImage(
            // Guid is assigned after image is added
            SerializableGuid.empty,
            // No texture associated with this reference image
            SerializableGuid.empty,
            sizeInMeters, "My Image", null);

        var jobHandle = mutableLibrary.ScheduleAddImageJob(
            grayscaleImageBytes,
            new Vector2Int(widthInPixels, heightInPixels),
            TextureFormat.R8,
            referenceImage);

        // Schedule a job that deallocates the image bytes after the image
        // is added to the reference image library.
        new DeallocateJob { data = grayscaleImageBytes }.Schedule(jobHandle);
    }
    else
    {
        // Cannot add the image, so dispose its memory.
        grayscaleImageBytes.Dispose();
    }
}

Multiple add image jobs can be processed concurrently. Whether or not MutableRuntimeReferenceImageLibrary is currently in use for image tracking has no effect on this.

Creating a manager at runtime

When you add a component to an active GameObject at runtime, Unity immediately invokes its OnEnable method. However, the ARTrackedImageManager requires a non-null reference image library. If the reference image library is null when the ARTrackedImageManager is enabled, it will automatically disable itself.

To add an ARTrackedImageManager at runtime, set its reference image library and then re-enable it:

var manager = gameObject.AddComponent<ARTrackedImageManager>();
manager.referenceLibrary = myLibrary;
manager.enabled = true;

Maximum number of moving images

Some providers can track moving images. This typically requires more CPU resources, so you can specify the number of moving images to track simultaneously. Check for support via the SubsystemDescriptor (ARTrackedImageManager.descriptor).

Tracked image prefab

This prefab is instantiated whenever an image from the reference image library is detected. The manager ensures the instantiated GameObject includes an ARTrackedImage component. You can get the reference image that was used to detect the ARTrackedImage with the ARTrackedImage.referenceImage property.

Tracking State

There are three possible tracking states for ARTrackedImages:

TrackingState	Description
None	The image is not being tracked. Note that this may be the initial state when the image is first detected.
Limited	The image is being tracked, but not as well. The situations in which an image is considered Limited instead of Tracking depend on the underlying AR framework. Examples that may cause Limited tracking include: Obscuring the image so that it is not visible to the camera. The image is not tracked as a moving image. This can happen, for example, if the maxNumberOfMovingImages is exceeded.
Tracking	The underlying AR SDK reports that it is actively tracking the image.

Determining when an image is visible

There is no API to determine the visibility of an image. Generally, if the tracking state is Tracking, it will likely change to Limited when the image is not visible. However, there are other situations in which the tracking state can be in states other than Tracked.

If this information is important to your application, considering comparing the ARTrackedImage's transform with the camera's view frustum.