Namespace UnityEngine.Perception.GroundTruth

Classes

AnimationPoseConfig

The animation pose config is a configuration file that maps a time range in an animation clip to a ground truth pose. The timestamp record is defined by a pose label and a start time. The timestamp records are order dependent.

BoundingBox2DLabeler

Produces 2d bounding box annotations for all visible objects each frame.

BoundingBox3DAnnotation

The generated product of the labeler

BoundingBox3DDefinition

The annotation definition of a 3D bounding box labeler

BoundingBox3DLabeler

Produces 3d bounding box ground truth for all visible and Labeling objects each frame.

BoundingBoxAnnotation

Bounding boxes for all of the labeled objects in a capture

BoundingBoxDefinition

The annotation definition of a bounding box labeler

CameraLabeler

Abstract class for defining custom annotation and metric generation to be run by PerceptionCamera. Instances of CameraLabeler on labelers will be invoked each frame the camera is set to capture data (see ShouldCaptureThisFrame).

DatasetCapture

Global manager for frame scheduling and output capture for simulations. Data capture follows the schema defined in TODO: Expose schema publicly

HUDPanel

Heads up display panel used to publish a key value pair on the screen. Items added to this need to have their values updated every frame, or else, they will be determined to be stale and removed from the view and re-used for a new entry.

IdLabelConfig

A definition for how a Labeling should be resolved to a single label and id for ground truth generation.

ImageEncoder

A utility for encoding raw pixel data into a variety of formats using the Unity Job System.

InstanceIdToColorMapping

Static class to procedurally generate a unique color for an instance ID. This algorithm is deterministic, and will always return the same color for a ID, and the same ID for a color. ID 0 is reserved to be an invalid ID and is mapped to color black (0,0,0,255). Invalid IDs always map to black, and black always maps to ID 0. In order to try to create visually contrasting colors for IDs, there are a subset of IDs reserved (1-65) to be generated by applying the golden ration to find the next color in the HSL spectrum. All of these colors, and only theses colors, will be in the alpha channel 255. After the first 65 IDs, the color will be determined by iterating through all available RGB values in the alpha channels from 264 - 1. Alpha channel 0 is marked as invalid. This service will support over 4 billion unique IDs => colors [(256^4) - (256*2) + 64]

InstanceSegmentationAnnotation

The instance segmentation image recorded for a capture. This includes the data that associates a pixel color to an object.

InstanceSegmentationDefinition

Annotation definition for an instance segmentation

InstanceSegmentationLabeler

Produces instance segmentation for each frame.

JointLabel

Label to designate a custom joint/keypoint. These are needed to add body parts to a humanoid model that are not contained in its

These label's can also be applied to the keypoints found in the to override the self occlusion tolerance values defined in the KeypointTemplate file. The KeypointTemplate defines the typical tolerances for a model, but certain models may need to have specific overrides on a keypoint for the self occlusion to work properly.

KeypointAnnotation

The product of the keypoint labeler

KeypointAnnotationDefinition

The definition of the keypoint

KeypointDefinition

A definition of a keypoint (joint).

KeypointLabeler

Produces keypoint annotations for a humanoid model. This labeler supports generic KeypointTemplate. Template values are mapped to rigged . Custom joints can be created by applying JointLabel to empty game objects at a body part's location.

Keypoints are recorded by this labeler with a state value describing if they are present on the model, present but not visible, or visible. A keypoint can be listed as not visible for three reasons: it is outside of the camera's view frustum, it is occluded by another object in the scene, or it is occluded by itself, for example a raised arm in front a model's face could occlude its eyes from being visible. To calculate self occlusion values, the keypoint labeler uses tolerances per keypoint to determine if the keypoint is blocked. The initial tolerance value for each keypoint is set per keypoint in the KeypointTemplate file. The tolerance of a custom keypoints can be set with the JointLabel used to create the keypoint. Finally, a KeypointOcclusionOverrides component be added to a model to apply a universal scaling override to all of the keypoint tolerances defined in a keypoint template.

KeypointOcclusionOverrides

When attached to a model used by the KeypointLabeler overrides the distance values for each keypoint defined in KeypointTemplate by multiplying them by this overrideDistanceScale scalar. The values in KeypointTemplate are generally set for a typical adult model, which makes it so that these values do not meet the needs of models with different body types (i.e. children, different heights, different weights). Changing the value of the scalar will help to get keypoint occlusion working properly for these models. A value of 1.0 will use the template values as is.

KeypointTemplate

Template used to define the keypoints of a humanoid asset.

KeypointValue

The value of an individual keypoint on a keypoint component.

LabelConfig<T>

A definition for how a Labeling should be resolved to a single label and id for ground truth generation.

LabelEntryMatchCache

Cache of instance id -> label entry index for a LabelConfig. This is not well optimized and is the source of a known memory leak for apps that create new instances frequently.

Labeling

Defines a set of labels associated with the object and its descendants. A Labeling component will override any Labeling components on the object's ancestors.

LabelManager

Manages the registration of Labeling components

ObjectCountLabeler

Labeler which produces object counts for each label in the associated IdLabelConfig each frame.

ObjectCountMetricDefinition

Metric definition for object counts

OverlayPanel

Some labeler's result in a full screen image per frame. The overlay panel controls which of these labeler's image is currently shown.

PerceptionCamera

Captures ground truth from the associated Camera.

PoseTimestampRecord

Record that maps a pose to a timestamp

RenderedObjectInfoGenerator

A CPU-based pass which computes bounding box and pixel counts per-object from instance segmentation images

RenderedObjectInfoLabeler

Labeler which produces label id, instance id, and visible pixel count in a single metric each frame for each object which takes up one or more pixels in the camera's frame.

RenderedObjectInfoMetricDefinition

Definition of the metric.

SemanticSegmentationAnnotation

SemanticSegmentationDefinition

Annotation definition for semantic segmentation

SemanticSegmentationLabelConfig

A definition for how a Labeling should be resolved to a single label and color for semantic segmentation generation.

SemanticSegmentationLabeler

Labeler which generates a semantic segmentation image each frame. Each object is rendered to the semantic segmentation image using the color associated with it based on the given SemanticSegmentationLabelConfig. Semantic segmentation images are saved to the dataset in PNG format. Only one SemanticSegmentationLabeler can render at once across all cameras.

SkeletonDefinition

A skeletal connection between two joints.

VisualizationHelper

Helper class that contains common visualization methods useful to ground truth labelers.

Structs

AnnotationHandle

A handle to an annotation. Can be used to report metrics on the annotation.

AsyncFuture<T>

A handle back to a result that will be calculated in the future. This class is used to report an asynchronous solution.

BoundingBox

BoundingBox3D

The specifics of each reported box

IdLabelConfig.LabelEntrySpec

A structure representing a label entry for writing out to datasets.

IdLabelEntry

An entry for IdLabelConfig mapping a label to an integer id.

InstanceSegmentationEntry

An instance segmentation entry

KeypointAnnotationDefinition.JointDefinition

KeypointAnnotationDefinition.SkeletonDefinition

KeypointAnnotationDefinition.Template

KeypointComponent

RenderedObjectInfo

Describes an instance of an object in an instance segmentation frame. Generated by RenderedObjectInfoGenerator.

SemanticSegmentationDefinitionEntry

SemanticSegmentationLabelEntry

LabelEntry for SemanticSegmentationLabelConfig. Maps a label to a color.

SemanticSegmentationLabeler.ImageReadbackEventArgs

Event information for imageReadback

SensorHandle

A handle to a sensor managed by the DatasetCapture. It can be used to check whether the sensor is expected to capture this frame and report captures, annotations, and metrics regarding the sensor.

Namespace UnityEngine.Perception.GroundTruth

Classes

Structs

Interfaces

Enums