AR Tracked Image Manager component
The tracked image manager is a type of trackable manager and performs 2D image tracking.
AR Tracked Image Manager component
The tracked image manager creates GameObjects for each detected image in the environment. Before an image can be detected, the manager must be instructed to look for a set of reference images compiled into a reference image library. It only detects images in this library.
Terminology
Term | Description |
---|---|
Reference image | A reference image is an image that the XRImageTrackingSubsystem attempts to find in the real world. The subsystem associates detected images with the reference image used to detect them. Each detected image has a pose in the world. |
Reference image library | A set of reference images. When you start an image tracking subsystem, you must first provide it with a library of reference images so it knows what to search for. |
Create a reference image library
You create reference image libraries in the Unity Editor. From Unity's main menu, go to Assets > Create > XR > Reference Image Library.
This creates a new reference image library asset in your project. To add images, select this Asset, then click Add Image and complete the fields that appear:
You can specify the following information:
Field | Description |
---|---|
Name | A string name that is available at runtime. This name isn't used by the subsystem, but it can be useful to identify which reference image has been detected. There is no check for duplicate name conflicts. |
Specify Size | If enabled, you can specify the physical size you expect the image to be in the real world. This field is mandatory for some providers and optional for others. For more information, see your provider's documentation. If you specify this field, the dimensions must be greater than zero. Editing one dimension (for example, width) causes the other dimension (height) to change automatically based on the image's aspect ratio. |
Keep Texture at Runtime | If enabled, XRReferenceImage.texture contains a reference to the source texture. This can be useful if you need access to the source texture at runtime. By default, this is unchecked to reduce the built Player size. When unchecked, XRReferenceImage.texture is null. |
Tip
You can include an XRReferenceImageLibrary
in an asset bundle and use it in an app that was not built with that reference image library present.
Reference image library
The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null. You can set it via script with:
ARTrackedImageManager manager = ...;
manager.referenceLibrary = myReferenceImageLibrary;
You can set the reference image library to be either an XRReferenceImageLibrary or a RuntimeReferenceImageLibrary. You can only create an XRReferenceImageLibrary
in the Editor, and you can't modify it at runtime. A RuntimeReferenceImageLibrary
is the runtime representation of an XRReferenceImageLibrary
. When you set the library to be an XRReferenceImageLibrary
, the image tracking subsystem automatically converts it to a RuntimeReferenceImageLibrary
for consumption.
The actual image library data is provider-specific; see your provider's documentation for details.
You can create a RuntimeReferenceImageLibrary
from an XRReferenceImageLibrary
with the ARTrackedImageManager.CreateRuntimeLibrary
method:
XRReferenceImageLibrary serializedLibrary = ...
RuntimeReferenceImageLibrary runtimeLibrary = trackedImageManager.CreateRuntimeLibrary(serializedLibrary);
Note
The ordering of the XRReferenceImages in the RuntimeReferenceImageLibrary
is undefined; that is, it might not match the order in which the images appeared in the source XRReferenceImageLibrary
. Each reference image does have a string name that you assign it, and a randomly assigned Guid. The Guid
are the same between the source XRReferenceImageLibrary
and its corresponding RuntimeReferenceImageLibrary
.
Use reference image libraries with asset bundles
Prior to AR Foundation 4.2, reference image libraries had to be built into the Player; that is, the XRReferenceImageLibrary
served only as a means to look up data that was expected to be packaged in the app. This meant that you could not, for example, download a novel reference image library to an already released app. As of AR Foundation 4.2, platform-specific data is attached to the XRReferenceImageLibrary
asset when building a Player or asset bundle. This means that you can include an XRReferenceImageLibrary
in an asset bundle and use it in an app that was not built with that reference image library present.
Respond to detected images
Subscribe to the ARTrackedImageManager's trackedImagesChanged event to be notified whenever an image is added (that is, first detected), updated, or removed:
[SerializeField]
ARTrackedImageManager m_TrackedImageManager;
void OnEnable() => m_TrackedImageManager.trackedImagesChanged += OnChanged;
void OnDisable() => m_TrackedImageManager.trackedImagesChanged -= OnChanged;
void OnChanged(ARTrackedImagesChangedEventArgs eventArgs)
{
foreach (var newImage in eventArgs.added)
{
// Handle added event
}
foreach (var updatedImage in eventArgs.updated)
{
// Handle updated event
}
foreach (var removedImage in eventArgs.removed)
{
// Handle removed event
}
}
Note that images also have a tracking state which can provide additional information about the tracking quality. An image that goes out of view, for example, might not be "removed", but its tracking state likely changes.
You can also get all the currently tracked images with the ARTrackedImageManager's trackables property. This acts like an IEnumerable collection, so you can use it in a foreach
statement:
void ListAllImages()
{
Debug.Log(
$"There are {m_TrackedImageManager.trackables.count} images being tracked.");
foreach (var trackedImage in m_TrackedImageManager.trackables)
{
Debug.Log($"Image: {trackedImage.referenceImage.name} is at " +
$"{trackedImage.transform.position}");
}
}
Or access a specific image by its TrackableId:
ARTrackedImage GetImageAt(TrackableId trackableId)
{
return m_TrackedImageManager.trackables[trackableId];
}
Tracked Image Prefab
The ARTrackedImageManager has a "Tracked Image Prefab" field; however, this is not intended for content. When an image is detected, ARFoundation creates a new GameObject to represent it.
If "Tracked Image Prefab" is null
, then AR Foundation creates a GameObject with an ARTrackedImage component on it. However, if you want every tracked image to also include additional components, you can provide a Prefab for AR Foundation to instantiate for each detected image. In other words, the purpose of the Prefab field is to extend the default behavior of tracked images; it is not the recommended way to place content in the world.
If you would like to instantiate content at the pose of the detected image and have its pose updated automatically, then you should parent your content to the ARTrackedImage
.
Add new reference images at runtime
Some subsystems might support image libraries that are modifiable at runtime. In this case, the subsystem produces a RuntimeReferenceImageLibrary
that is a MutableRuntimeReferenceImageLibrary
. To use it, you need to cast the RuntimeReferenceImageLibrary
to a MutableRuntimeReferenceImageLibrary
:
[SerializeField]
ARTrackedImageManager m_TrackedImageManager;
void AddImage(Texture2D imageToAdd)
{
if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
return; // Session state is invalid
if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
{
mutableLibrary.ScheduleAddImageWithValidationJob(
imageToAdd,
"my new image",
0.5f /* 50 cm */);
}
}
To create an empty library that you can add images to later, you can call CreateRuntimeLibrary
without arguments:
void AddImage(Texture2D imageToAdd)
{
if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
return; // Session state is invalid
var library = m_TrackedImageManager.CreateRuntimeLibrary();
if (library is MutableRuntimeReferenceImageLibrary mutableLibrary)
{
mutableLibrary.ScheduleAddImageWithValidationJob(
imageToAdd,
"my new image",
0.5f /* 50 cm */);
}
}
You can check whether a particular tracked image manager supports mutable libraries with its descriptor:
bool DoesSupportMutableImageLibraries()
{
return m_TrackedImageManager.descriptor.supportsMutableLibrary;
}
You can add images to mutable libraries allow images at any time. Adding an image can be computationally resource-intensive, and might take a few frames to complete. The Unity Job System is used to process images asynchronously.
To add an image to a MutableRuntimeReferenceImageLibrary, use the ScheduleAddImageJob method. This returns a JobHandle that you can use to determine when the job is complete. You can safely discard this handle if you don't need to do this.
If you use the extension method which accepts a Texture2D, you do not need to manage any memory.
If you use the version of ScheduleAddImageJob that accepts a NativeSlice or pointer, you are responsible for managing the memory, that is, freeing it when the job completes. You can do this by scheduling a dependent job that frees the memory:
struct DeallocateJob : IJob
{
[DeallocateOnJobCompletion]
public NativeArray<byte> data;
public void Execute() { }
}
void AddImage(NativeArray<byte> grayscaleImageBytes,
int widthInPixels, int heightInPixels,
float widthInMeters)
{
if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
return; // Session state is invalid
if (m_TrackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
{
var aspectRatio = (float)widthInPixels / (float)heightInPixels;
var sizeInMeters = new Vector2(widthInMeters, widthInMeters * aspectRatio);
var referenceImage = new XRReferenceImage(
// Guid is assigned after image is added
SerializableGuid.empty,
// No texture associated with this reference image
SerializableGuid.empty,
sizeInMeters, "My Image", null);
var jobState = mutableLibrary.ScheduleAddImageWithValidationJob(
grayscaleImageBytes,
new Vector2Int(widthInPixels, heightInPixels),
TextureFormat.R8,
referenceImage);
// Schedule a job that deallocates the image bytes after the image
// is added to the reference image library.
new DeallocateJob { data = grayscaleImageBytes }.Schedule(jobState.jobHandle);
}
else
{
// Cannot add the image, so dispose its memory.
grayscaleImageBytes.Dispose();
}
}
Multiple add image jobs can be processed concurrently. Whether or not MutableRuntimeReferenceImageLibrary
is currently in use for image tracking has no effect on this.
Reference images and texture import settings
Image detection algorithms rely on accurate knowledge of a reference image's aspect ratio; however, Unity's default texture import setting adjusts texture resolutions to the nearest powers of 2 (PoT). This can have a negative effect on tracking capabilities when tracking non-power of 2 (NPoT) images.
When a NPoT image is imported with default settings and added to a MutableRuntimeReferenceImageLibrary
during runtime, it may display squashing, stretching, or scaling artifacts.
To ensure images maintain their aspect ratio when imported, select the image in your project assets and change the Non-Power of 2 texture import setting to None.
Note
For textures with power-of 2 aspect ratios, Non-Power of 2 setting will not be selectable as no adjustments are necessary.
Create a manager at runtime
When you add a component to an active GameObject
at runtime, Unity immediately invokes its OnEnable
method. However, the ARTrackedImageManager
requires a non-null reference image library. If the reference image library is null when the ARTrackedImageManager
is enabled, it automatically disables itself.
To add an ARTrackedImageManager
at runtime, set its reference image library and then re-enable it:
var manager = gameObject.AddComponent<ARTrackedImageManager>();
manager.referenceLibrary = myLibrary;
manager.enabled = true;
Maximum number of moving images
Some providers can track moving images. This typically requires more CPU resources, so you can specify the number of moving images to track simultaneously. Check for support via the SubsystemDescriptor
(ARTrackedImageManager.descriptor
).
Tracked Image Prefab
This Prefab is instantiated whenever an image from the reference image library is detected. The manager ensures the instantiated GameObject
includes an ARTrackedImage
component. You can get the reference image that was used to detect the ARTrackedImage
with the ARTrackedImage.referenceImage
property.
Tracking State
There are three possible tracking states for ARTrackedImages
:
TrackingState | Description |
---|---|
None | The image is not being tracked. Note that this may be the initial state when the image is first detected. |
Limited | The image is being tracked, but not as effectively. The situations in which an image is considered Limited instead of Tracking depend on the underlying AR framework. Examples that could cause Limited tracking include:
|
Tracking | The underlying AR SDK reports that it is actively tracking the image. |
Determining when an image is visible
There is no API to determine the visibility of an image. Generally, if the tracking state is Tracking, it likely changes to Limited
when the image is not visible. However, there are other situations in which the tracking state can be in states other than Tracked
.
If this information is important to your application, considering comparing the ARTrackedImage
's transform with the camera's view frustum.