DatasetCapture

DatasetCapture tracks egos, sensors, annotations, and metrics, combining them into a unified JSON-based dataset on disk. It also controls the simulation time elapsed per frame to accommodate the active sensors.

Sensor scheduling

While sensors are registered, DatasetCapture ensures that frame timing is deterministic and run at the appropriate simulation times to let each sensor run at its own rate.

Using Time.CaptureDeltaTime, it also decouples wall clock time from simulation time, allowing the simulation to run as fast as possible.

Custom sensors

Custom sensors can be registered using DatasetCapture.RegisterSensor(). The period passed in at registration time determines how often in simulation time frames should be scheduled for the sensor to run. The sensor implementation would then check ShouldCaptureThisFrame on the returned SensorHandle each frame to determine whether it is time for the sensor to perform a capture. SensorHandle.ReportCapture should then be called in each of these frames to report the state of the sensor to populate the dataset.

Custom annotations and metrics

In addition to the common annotations and metrics produced by PerceptionCamera, scripts can produce their own via DatasetCapture. Annotation and metric definitions must first be registered using DatasetCapture.RegisterAnnotationDefinition() or DatasetCapture.RegisterMetricDefinition(). These return AnnotationDefinition and MetricDefinition instances which can then be used to report values during runtime.

Annotations and metrics are always associated with the frame they are reported in. They may also be associated with a specific sensor by using the Report* methods on SensorHandle.

Example

using System;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;

[RequireComponent(typeof(PerceptionCamera))]
public class CustomAnnotationAndMetricReporter : MonoBehaviour
{
    public GameObject light;
    public GameObject target;

    MetricDefinition lightMetricDefinition;
    AnnotationDefinition boundingBoxAnnotationDefinition;
    SensorHandle cameraSensorHandle;

    public void Start()
    {
        //Metrics and annotations are registered up-front
        lightMetricDefinition = DatasetCapture.RegisterMetricDefinition(
            "Light position",
            "The world-space position of the light",
            Guid.Parse("1F6BFF46-F884-4CC5-A878-DB987278FE35"));
        boundingBoxAnnotationDefinition = DatasetCapture.RegisterAnnotationDefinition(
            "Target bounding box",
            "The position of the target in the camera's local space",
            id: Guid.Parse("C0B4A22C-0420-4D9F-BAFC-954B8F7B35A7"));
    }

    public void Update()
    {
        //Report the light's position by manually creating the json array string.
        var lightPos = light.transform.position;
        DatasetCapture.ReportMetric(lightMetricDefinition,
            $@"[{{ ""x"": {lightPos.x}, ""y"": {lightPos.y}, ""z"": {lightPos.z} }}]");
        //compute the location of the object in the camera's local space
        Vector3 targetPos = transform.worldToLocalMatrix * target.transform.position;
        //Report using the PerceptionCamera's SensorHandle if scheduled this frame
        var sensorHandle = GetComponent<PerceptionCamera>().SensorHandle;
        if (sensorHandle.ShouldCaptureThisFrame)
        {
            sensorHandle.ReportAnnotationValues(
                boundingBoxAnnotationDefinition,
                new[] { targetPos });
        }
    }
}

// Example metric that is added each frame in the dataset:
// {
//   "capture_id": null,
//   "annotation_id": null,
//   "sequence_id": "9768671e-acea-4c9e-a670-0f2dba5afe12",
//   "step": 1,
//   "metric_definition": "1f6bff46-f884-4cc5-a878-db987278fe35",
//   "values": [{ "x": 96.1856, "y": 192.676, "z": -193.8386 }]
// },

// Example annotation that is added to each capture in the dataset:
// {
//   "id": "33f5a3aa-3e5e-48f1-8339-6cbd64ed4562",
//   "annotation_definition": "c0b4a22c-0420-4d9f-bafc-954b8f7b35a7",
//   "values": [
//     [
//       -1.03097284,
//       0.07265166,
//       -6.318692
//     ]
//   ]
// }