What is an RGB camera?
An RGB camera captures images of the scene and publishes them as either uncompressed RGB images or compressed images with JPG or PNG encodings.
Sensor Output
Schema for uncompressed images
# An uncompressed image.
# (0, 0) is at top-left corner of image
# +x point to the right in the image
# +y point down in the image
# +z points into to plane of the image
Header header
uint32 seq
time stamp # the acquisition time of the image.
string frame_id # the optical frame of camera
uint32 height # image height, that is, number of rows
uint32 width # image width, that is, number of columns
string encoding # Encoding of pixels is always "rgba8":
# 4x 8-bit channels: red, green, blue, alpha
uint8 is_bigendian # is this data big-endian?
uint32 step # Full row length in bytes (width)
uint8[] data # actual matrix data, size = step * rows (i.e. width * height)
Unity Simulation ImageMsg Structure/Schema
public class ImageMsg : IMessage
{
public const string k_MessageTypeName = "sensor_msgs/Image";
public string MessageTypeName => k_MessageTypeName;
// This message contains an uncompressed image
// (0, 0) is at top-left corner of image
// +x should point to the right in the image
// +y should point down in the image
// +z should point into to plane of the image
public Std.HeaderMsg header;
// Header timestamp should be acquisition time of image
// Header frame_id should be optical frame of camera
// origin of frame should be optical center of cameara
public uint height; // image height, that is, number of rows
public uint width; // image width, that is, number of columns
public string encoding; // Encoding of pixels is always "rgba8":
// 4x 8-bit channels: red, green, blue, alpha
public byte is_bigendian; // is this data bigendian?
public uint step; // Full row length in bytes
public byte[] data; // actual matrix data, size is (step * rows)
}
Schema for compressed images
**ROS Compressed Image Message Schema
# This message contains a compressed image
Header header # Header timestamp should be acquisition time of image
# Header frame_id should be optical frame of camera
# origin of frame should be optical center of camera
# +x should point to the right in the image
# +y should point down in the image
# +z should point into to plane of the image
string format # Specifies the format of the data
# Acceptable values:
# jpeg, png
uint8[] data # Compressed image buffer
Unity Simulation CompressedImageMsg Structure/Schema
public class CompressedImageMsg : IMessage
{
public const string k_MessageTypeName = "sensor_msgs/CompressedImage";
public string MessageTypeName => k_MessageTypeName;
// This message contains a compressed image.
public StdMessages.HeaderMsg header;
// Header timestamp should be acquisition time of image
// Header frame_id should be optical frame of camera
// origin of frame should be optical center of cameara
// +x should point to the right in the image
// +y should point down in the image
// +z should point into to plane of the image
public string format;
// Specifies the format of the data
// Acceptable values:
// jpeg, png
public byte[] data;
// Compressed image buffer
}
Compressed image output
By default, the RGB Camera outputs uncompressed images. JPG or PNG Compression can be enabled in the Inspector under Compression Parameters.
Note
For visualization to work in RVIZ, for compressed images, the topic name must end with /compressed.
RGB camera model
The RGB camera is packaged within the CameraSensor.
Sensor model implications
CameraSensor uses Unity's Camera and HD Camera components and offers a lot of configurability.
Most camera specific configurations can be directly set in the properties of the Camera and HD Camera in the Inspector. However the values of the following parameters will be overwritten by the values set through the properties under the CameraSensor component: fov, near clip, far clip.
Distortion:
Currently camera distortion (other than Brown Conrady, see below) can only be implemented by adding a distortion effect to a Volume.
Brown Conrady Distortion
This applies a post processing effect on the camera. The implementation is a Brown-Conrady model used to calculate the distorted pixels from undistorted pixels. An example of the distortion can be seen below.
The following formula is used:
with the following variables:
is the center of distortion;
is the undistorted point;
is the distorted point;
is the distance to the center of distortion;
are the radial distortion parameters;
are the tangential distortion parameters;
represents the following function:
.
refer to the 
and 
components of , respectively;
refer to the and components of , respectively;
refer to the and components of , respectively
To use this distortion model, change the Distortion Parameters in the inspector within the CameraNode component, as shown below. The parameters apply to both RGB and Depth images. Leaving all parameters set to 0 will cause no distortion.
Examples of distortion can be seen below. A negative K1 value results in a pincushion distortion, while a positive value gives a barrel distortion. Typical (absolute) values are less than 1.
Noise
By default, the RGB camera sensor will report the "actual" color per pixel captured in the Unity camera's render texture.
We support adding noise sampled from a Gaussian distribution independent to each channel of each pixel. This can be enabled by setting the camera noise type parameter to gaussian and specifying the camera noise mean and standard deviation.
Grayscale Filtering
By default, the RGB camera sensor will render all channels of color. A grayscale version can be rendered instead by filtering, either by one of the color channels (red, green or blue) or by luminance. To do this, change the "Filter" under "grayscale Filtering" in the Inspector of the Camera Node. Note that this feature generates a monochromatic image where the brightness represents the intensity of the color channel chosen. It does not return an RGB image.
Examples of filtering can be seen below.
Properties
| Property | Function | XML Element |
|---|---|---|
| Update Rate | The update rate in hz is the frequency at which the sensor will generate data. |
sensor/update_rate |
| Offset Start Time | Determines if the sensor should offset its start time by a small, random number of frames. | sensor/offset_start_time |
| Camera | The camera component used to capture data. | |
| Horizontal FOV | The horizontal field of view of the camera (radians in the description file, degrees in the Unity Inspector). | sensor/camera/horizontal_fov |
| Image Width | The image width in pixels. | sensor/camera/image/width |
| Image Height | The image height in pixels. | sensor/camera/image/height |
| Clip Near | Near clipping plane of the camera. | sensor/camera/clip/near |
| Clip Far | Far clipping plane of the camera. | sensor/camera/clip/far |
| Enable Depth Output | Enable depth image output. | sensor/depth/camera_info/enable |
| Enable Camera Info Output | Enable camera information output. | sensor/camera_info/enable |
| Enable Camera Info Depth Output | Enable camera depth information output. | sensor/depth/camera_info/enable |
| Render Every Frame | Whether to render the camera automatically each unity update. Publishing will still happen at the update rate. | |
| Camera Noise Type | The type of noise to generate onto within the RGB output. | sensor/camera/noise/type |
| Camera Noise Mean | The mean of the Gaussian distribution from which additive noise values are drawn within the RGB output. | sensor/camera/noise/mean |
| Camera Noise Stddev | The standard deviation of the Gaussian distribution from which additive noise values are drawn within the RGB output. | sensor/camera/noise/stddev |
| Depth Camera Noise Type | The type of noise to generate onto within the depth output. | sensor/depth/noise/type |
| Depth Camera Noise Mean | The mean of the Gaussian distribution from which additive noise values are drawn within the depth output. | sensor/depth/noise/mean |
| Depth Camera Noise Stddev | The standard deviation of the Gaussian distribution from which additive noise values are drawn within the depth output. | sensor/depth/noise/stddev |
| Image Publisher Topic | The name of the topic on which RGB image data is published. | sensor/topic |
| Depth Publisher Topic | The name of the topic on which depth image data is published. | sensor/depth/topic |
| Camera Info Publisher Topic | The name of the topic on which camera info data is published. | sensor/camera/camera_info/topic |
| Depth Camera Info Publisher Topic | The name of the topic on which depth camera info data is published. | sensor/depth/camera_info/topic |
Import/Export example
You can import and export a camera sensor from an XML format similar to the example below:
<sensor name="rgb-camera" type="camera">
<update_rate>30</update_rate>
<topic>/unity/rgb</topic>
<camera>
<horizontal_fov>1.047</horizontal_fov>
<image>
<width>256</width>
<height>256</height>
</image>
<clip>
<near>0.1</near>
<far>100</far>
</clip>
<noise>
<type>gaussian</type>
<mean>0.0</mean>
<stddev>0.02</stddev>
</noise>
</camera>
</sensor>