Use point cloud data

Many sensors, such as lidars and stereoscopic cameras, generate a collection of 3D points in space known as a point cloud. In SensorSDK, the PhotosensorEncoder/PhotosensorToPointCloud node manages the creation of the PointCloud data structure coming from the photosensor.

Point cloud data format

The PointCloud class stores the point cloud in a compute buffer, with a collection of PointXYZI structures. The PointXYZI structures contain the following:

• Vector3 position: the point cartesian coordinates in the sensor space.
• float intensity: the intensity perceived by the photosensor.

If the photosensor doesn't perceive enough light in a given direction or the point is out of range, a point with values (0, 0, 0, 0) is still generated in the PointCloud, indicating an invalid point.

Retrieving point cloud data from the GPU

When working with point clouds, keep the data as long as possible on the GPU and avoid memory transfers to improve performance. Retrieving the point cloud from the GPU is an asynchronous operation that might cause latency.

Here is a minimal SystemGraph node that demonstrates how to retrieve the point cloud data from GPU memory:

using System.Linq;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.Rendering.HighDefinition;
using Mechatronics.SystemGraph;
using Mechatronics.SensorSDK;

[NodeCategory("Debug", "Get Point Cloud", NodeTick.Synchronous)]
public class GetPointCloud : NodeRuntime
{
    [Tooltip("Point cloud to get.")]
    [Field("PointCloud", PortDirection.Left, FieldExtra.Read)]
    public PortType<PointCloud> pointCloud = new();

    [Tooltip("Graphic context used to transcode data.")]
    [Field("InTranscode", PortDirection.Left, FieldExtra.Read | FieldExtra.ChangeEvent)]
    public PortType<CustomPassContext> inTranscode = new();

    [Tooltip("Number of valid points in the point cloud.")]
    [Field("NumValidPoints", PortDirection.Right, FieldExtra.Write)]
    public PortType<int> numValidPoints = new();

    private List<PointXYZI> _points = new();
    private bool _newPointCloud;

    public override void Enable(Scheduler.ClockState clockState)
    {
        // By convention, transcode signal is written last, so register callback when
        // the transcode signal changes.
        inTranscode.ChangeEvent += OnTranscode;
        _newPointCloud = false;
    }

    public override void Disable()
    {
        inTranscode.ChangeEvent -= OnTranscode;
    }

    public override bool OnTick(double now, double eventTime, Scheduler.ClockState clockState, Scheduler.Signal signal)
    {
        if (_newPointCloud)
        {
            // Process the point cloud during the next synchronous operation.
            int numValid = _points.Where(point => point.IsValid).Count();
            _newPointCloud = false;

            // Write to outputs.
            numValidPoints.Write = numValid;
        }
        return true;
    }

    // Called when InTranscode changes
    private void OnTranscode()
    {
        CommandBuffer cmd = inTranscode.Read.cmd;
        PointCloud cloud = pointCloud.Read;
        if (cmd != null && cloud != null)
        {
            // It is better practice to keep readback callbacks as fast and predictable
            // as possible. Points passed to the callback are only valid during the
            // callback, so they must be copied if they are to be used in OnTick. Avoid
            // writing to the node outputs inside readback callbacks.
            cmd.RequestAsyncReadback(cloud, points =>
            {
                _points.Clear();
                _points.AddRange(points);
                _newPointCloud = true;
            });
        }
    }
}

Relevant nodes

System graph has the following nodes to use with point cloud data:

Viewer/Point cloud viewer: Displays the acquired point cloud as particles in the scene.

Transcoder/PointCloudToDepth: Generates a perspective depth map texture from the point cloud.

Connectivity/PointCloudToFile: Saves the point cloud to disk in either LAS or PCD format.