About ARCore XR Plug-in

The ARCore XR Plug-in package enables ARCore support via Unity's multi-platform XR API. This package implements the following XR subsystems:

• Session
• Camera
• Depth
• Input
• Planes
• Raycast
• Anchors
• Face tracking
• Image tracking
• Environment probes
• Occlusion

This version of ARCore XR Plug-in uses ARCore 1.24 and supports the following functionality:

• Device localization
• Horizontal plane detection
• Vertical plane detection
• Point clouds
• Pass-through camera view
• Light estimation
• Anchors
• Oriented feature points
• Hit testing
• Session management
• ARCore APK on-demand installation
• 2D Image tracking
• Face tracking
• Environment probes
• Occlusion

This package does not support the following subsystems:

• Object tracking
• Participant
• Mesh
• Body tracking

Installing ARCore XR Plug-in

To install this package, follow the instructions in the Package Manager documentation.

You can also install the AR Foundation package, which uses the ARCore XR Plug-in and provides many useful scripts and prefabs. For more information, see the AR Foundation documentation.

Using ARCore XR Plug-in

In most cases, you should use the scripts, prefabs, and assets provided by the AR Foundation package as the basis for your handheld AR apps rather than using ARCore APIs directly. The ARCore XR plug-in supports AR Foundation features on the Android platform by implementing the native endpoints required to target Google’s ARCore platform using Unity's multi-platform XR API.

Use the ARCore XR plug-in APIs when you need access to Android ARCore-specific features. The ARCoreFaceRegions sample in the AR Foundations repository provides an example of using an ARCore feature.

See Using AR Foundation for general information about developing AR apps in Unity.

Build Settings

You can flag the ARCore XR Plug-in as Required or Optional in the Project Settings window.

ARCore is set to Required by default. If you set the ARCore plug-in to Optional instead, the Google Play store lets users install your app on devices that don't support ARCore, or devices that support ARCore but don't have it installed. Use the Optional setting when creating an app that provides different experiences depending on whether ARCore is available.

To create an ARCoreSettings Asset and assign it to your build settings, open the Project Settings window in Edit > Project Settings, then navigate to the XR Plug-in Management menu and enable the ARCore plug-in provider:

When you enable this setting, Unity creates an ARCoreSettings asset that you can access in the XR Plug-in Management > ARCore settings:

Note

If you set ARCore as Required and install your app on a device that does not support ARCore -- which you can do using Unity's Build and Run feature or by "side-loading" via USB -- the device will incorrectly report that ARCore is available. (This is because the Google Play Store prevents the installation of apps that require ARCore on unsupported devices, so these apps always assume they're running on a supported device.)

Session

ARCore implements XRSessionSubsystem.GetAvailabilityAsync. Use this function to determine if the device ARCore is currently running on is supported. ARCore's list of supported devices is frequently updated to include additional devices. For a full list of devices that ARCore supports, see ARCore supported devices.

If ARCore isn't already installed on a device, your app needs to check with the Google Play store to see if there's a version of ARCore that supports that device. To do this, use GetAvailabilityAsync to return a Promise that you can use in a coroutine. For ARCore, this check can take some time.

If the device is supported, but ARCore is not installed or requires an update, call XRSessionSubsystem.InstallAsync, which also returns a Promise.

For more information, see ARSubsystems session documentation.

Depth subsystem

Ray casts return a Pose for the item the ray cast hits. When you use a ray cast against feature points, the pose orientation provides an estimate for the surface the feature point might represent.

The depth subsystem doesn't require any additional resources, so it doesn't affect performance.

ARCore's depth subsystem only produces one XRPointCloud.

For more information, see ARSubsystems depth subsystem.

Plane tracking

ARCore supports plane subsumption. This means that you can include a plane inside another. Unity keeps the included (subsumed) plane and doesn't update it.

ARCore provides boundary points for all its planes.

The ARCore plane subsystem requires additional CPU resources and can use a lot of energy. The horizontal and vertical plane detection require additional resources when enabled. To save energy, disable plane detection when your app doesn't need it.

Setting the plane detection mode to PlaneDetectionMode.None works in the same way as using Stop for a subsystem.

For more information, see the ARSubsystems plane subsystem documentation.

Image tracking

To use image tracking on ARCore, you must create a reference image library. To do this, see the AR Subsystems documentation on image tracking.

When you build the Player for Android, the ARCore build code creates a imgdb file for each reference image library. ARCore creates these files in your project's StreamingAssets folder, in a subdirectory called HiddenARCore, so Unity can access them at runtime.

You can use .jpg or .png files as AR reference images in ARCore. If a reference image in the XRReferenceImageLibrary isn't a .jpg or .png, the ARCore build processor attempts to convert the Texture to a .png so that ARCore can use it.

When you export a Texture2D to .png, it can fail if the Texture's Texture Import Settings have Read/Write Enabled disabled and Compression is set to None.

To use the Texture at runtime (not as a source Asset for the reference image), create a separate .jpg or .png copy for the source Asset. This reduces the performance impact of the Texture Import Settings at runtime.

Reference image dimensions

To improve image detection in ARCore you can specify the image dimensions. When you specify the dimensions for a reference image, ARCore receives the image's width, and then determines the height from the image's aspect ratio.

Face tracking

For information about face tracking, see the ARSubsystems Face Tracking documentation.

The ARCore face subsystem provides face tracking methods that allow access to "regions". Regions are specific to ARCore. ARCore provides access to the following regions that define features on a face:

• Nose tip
• Forehead left
• Forehead right

Each region has a Pose associated with it. To access face regions, obtain an instance of the ARCoreFaceSubsystem using the following script:

XRFaceSubsystem faceSubsystem = ...
#if UNITY_ANDROID
var arcoreFaceSubsystem = faceSubsystem as ARCoreFaceSubsystem;
if (arcoreFaceSubsystem != null)
{
    var regionData = new NativeArray<ARCoreFaceRegionData>(0, Allocator.Temp);
    arcoreFaceSubsystem.GetRegionPoses(faceId, Allocator.Temp, ref regionData);
    using (regionData)
    {
        foreach (var data in regionData)
        {
            Debug.LogFormat("Region {0} is at {1}", data.region, data.pose);
        }
    }
}
#endif

Light estimation

ARCore light estimation can operate in two modes:

• LightEstimationMode.AmbientIntensity: Providers color correction and average pixel intensity information.
• LightEstimationMode.EnvironmentalHDR: Provides an estimated Main Light Direction, HDR Color, and the ambient SphericalHarmonicsL2 (see SphericalHarmonicsL2 for more information on Spherical Harmonics).

You can’t use both modes at the same time.

ARCore's Face tracking and Environment probes use or affect the light estimation mode. If one or both of these subsystems is present and enabled, it changes the light estimation mode behavior depending on the configuration:

Functionality	Supported light estimation modes	Modifiable
Face tracking	LightEstimationMode.AmbientIntensity, LightEstimationMode.Disabled	Yes
Environment probes	LightEstimationMode.EnvironmentalHDR	No

• Face tracking: ARCore doesn't support LightEstimationMode.EnvironmentalHDR when face tracking is enabled and rendering won't work when this mode is specified. To prevent errors, you can only set LightEstimationMode.AmbientIntensity or LightEstimationMode.Disabled when face tracking is enabled, or ARCore enforces LightEstimationMode.Disabled.

• Environment probes: For ARCore environment probes to update the cubemap, the light estimation mode must be set to LightEstimationMode.EnvironmentalHDR. This also allows ARCore to take ownership of the setting.

Camera configuration

XRCameraConfiguration contains an IntPtr field nativeConfigurationHandle, which is a platform-specific handle. For ARCore, this handle is the pointer to the ArCameraConfiguration. The native object is managed by Unity. Do not manually destroy it.

Occlusion

The ARCore implementation of XROcclusionSubsystem supports Environment Depth Texture but does not support the other Textures related to human segmentation.

Recording and playback

ARCore allows you to record an ArSession to an .mp4 and play it back at a later time. To support this feature, the ARCoreSessionSubsystem exposes the following methods:

• StartRecording
• StopRecording
• StartPlayback
• StopPlayback

To start a recording, supply an ArRecordingConfig. This specifies the file name that Unity saves the recording as, as well as other options. Call StopRecording to stop recording. When Unity stops recording, it creates the .mp4 file as specified in the ArRecordingConfig. This contains the camera feed and sensor data required by ARCore.

To play back a video, use the StartPlayback method, and specify an .mp4 file created during an earlier recording.

To start or stop a recorded file in ARCore, the ARCoreSessionSubsystem pauses the session. Pausing and resuming a session can take between 0.5 and 1.0 seconds.

Note: Video recordings contain sensor data, but not the computed results. ARCore does not always produce the same output, which means trackables might not be consistent between playbacks of the same recording. For example, multiple playbacks of the same recording might give different plane detection results.

Technical details

Requirements

This version of ARCore XR Plug-in is compatible with the following versions of the Unity Editor:

• 2020.3
• 2021.1
• 2021.2

Known limitations

• Color Temperature in degrees Kelvin is not currently supported.
• The XROcclusionSubsystemDescriptor properties supportsEnvironmentDepthImage and supportsEnvironmentDepthConfidenceImage require a session before support can be determined. If there is no session, then these properties return false. They might return true later when a session has been established.

Package contents

This version of ARCore XR Plug-in includes:

• A shared library which provides implementation of the XR Subsystems listed above
• A shader used for rendering the camera image
• A plug-in metadata file