What is a fisheye camera?
A fisheye camera uses an ultra wide-angle lens to capture images of the scene and publishes them as uncompressed RGB images.
Sensor Output
Schema
The fisheye camera uses the same uncompressed or compressed image messages as the RGB camera.
Compressed image output
The fisheye camera supports the same image compression parameters as the RGB camera.
Fisheye camera model
Implementation in Unity
The fisheye camera is implemented using multiple Unity cameras oriented in such a way that their combined field of view spans the desired fisheye lens field of view. A compute shader samples the camera textures and combines them into a single image, according to the specified fisheye parameters.
Fisheye variants
You can use one of two implementation variants:
- FisheyeCameraSensorCube: a 5-cameras implementation
- FisheyeCameraSensorLR: a 2-cameras implementation
The 5-cameras implementation covers the whole field of view, but is slower. The 2-cameras implementation covers the whole horizontal field of view, but is missing parts of the vertical field of view.
Projection model
Instead of using regular perspective projection (also called rectilinear projection), the fisheye camera uses equidistant projection, where distance from the image center is directly proportional to the angle the light ray forms with the camera Z axis.
Since the angle 𝜃 is in radians, the focal length is in pixels per radian. For example, a focal length of 300 indicates that a light ray forming an angle of 1 radian with the Z axis appears at a distance of 300 pixels from the fisheye center.
Distortion model
While the fisheye camera performs equidistant projection by default, it's possible to use other mapping functions, like stereographic or equisolid projections, by adding radial distortion to the basic projection. Refer to Wikipedia to learn more on fisheye lenses and mapping functions.
The fisheye camera uses the same camera intrinsic parameters as OpenCV. Refer to OpenCV for a description of the distortion model. Distortion parameters k1 through k4 are used to compute a distorted radial distance using the following formula:
Set all distortion parameters to 0 if you want to eliminate radial distortion and use a pure equidistant projection.
Important
To be compatible with OpenCV's fisheye model, the maximum field of view angle is 180 degrees.
Calibration
You can use the camera calibration sample scene to generate images of a calibration grid and the necessary XML files to use OpenCV to perform camera calibration. Since the fisheye camera uses the same projection and distortion models as OpenCV, you should get approximately the same camera intrinsic parameters from OpenCV as those you set in the Inspector. Note that the distortion parameters k1 through k4 are very sensitive by nature, and usually differ significantly more.
Sensor model implications
The fisheye camera Prefab has multiple child GameObjects, each one with a Unity camera. You can configure those cameras as you would any regular Unity camera. Their values for the clipping planes are overwritten by the values you set in the FisheyeCameraSensor
component.
Important
You shouldn't change the child cameras position, rotation or field of view, as these values have been chosen so that the combined field of view of all cameras cover the fisheye lens field of view.
Automatic exposure in HDRP
Since the fisheye camera sensor uses multiple Unity cameras, they must use the same exposure setting. Otherwise, seams can appear in the fisheye image. If you are using HDRP, each camera's exposure is adjusted automatically according to the range of brightness levels in its field of view, resulting in different exposures and visible seams.
By default, the fisheye camera sensor fixes this issue by adding a volume override around the sensor with a fixed exposure.
You can simulate Auto Exposure by setting the Exposure Mode to Simulate Auto Exposure. This will add an extra camera to the fisheye sensor that has an Auto Exposure volume and applies the automatically calculated exposure value to the fixed exposure.
If you want to configure the volume exposure override yourself, you can do the following:
- Select the fisheye camera in the Hierarchy window.
- In the Inspector window, set Exposure Mode to Default Auto Exposure.
- In the Project window, locate the DisableAutoExposureHDRP Prefab under
Packages\Simulation Sensors\Resources\Simulation\Sensors
. - Add the Prefab as a child of the fisheye sensor by dragging the Prefab to the Hierarchy window.
- Select the DisableAutoExposureHDRP Prefab in the Hierarchy window and locate its Volume component In the Inspector window.
- Select Clone next to the Profile property.
- Expand Exposure if necessary, then set Fixed Exposure to an appropriate value for your scene.
It's recommended to clone the volume profile as described above so that each fisheye camera has its own exposure setting, instead of sharing the same setting.
Alternatively, you can set a global volume on a single object in your scene to override the exposure settings. Note that using a global volume affects all cameras in the scene, including the main camera and other camera sensors from the Simulation package.
Properties
Property | Function | XML Element |
---|---|---|
Update Rate | The update rate in hz is the frequency at which the sensor will generate data. |
sensor/update_rate |
Offset Start Time | Determines if the sensor should offset its start time by a small, random number of frames. | sensor/offset_start_time |
Render Every Frame | Whether to render the camera automatically each unity update. Publishing will still happen at the update rate. | |
Topic | The name of the topic on which RGB image data is published. | sensor/topic |
Cameras | The list of camera components used to capture data. | |
Output Format | The fisheye image format: color or grayscale. Corresponds to "rgba8" and "mono8" respectively in the XML. | sensor/camera/image/format |
Output Texture Size | The fisheye image width and height in pixels. Reducing this will crop the image, effectively reducing the field of view. If you want to downsample the image instead, you must also apply the same multiplicative factor to the fisheye center and focal length. | sensor/camera/image/width sensor/camera/image/height |
Cubemap Texture Size | The internal camera image width and height in pixels. | |
Fisheye Center | The lens principal point, from the top-left of the fisheye image, in pixels. In the XML, both values are separated by a space. | sensor/camera/distortion/center |
Fisheye Focal | The lens focal length. X and Y values are in pixels per radian. In the XML, both values are separated by a space. | sensor/camera/focal |
Fisheye Distortion | The array of radial distortion coefficients k1 through k4. The array must be of size 4. | sensor/camera/distortion/k1 sensor/camera/distortion/k2 sensor/camera/distortion/k3 sensor/camera/distortion/k4 |
Horizontal FOV | The horizontal field of view of the camera. Only affects the maximum visible field of view, has no effect on projection. The angle is in degrees in the Unity Inspector, but it is in radians in the XML. | sensor/camera/horizontal_fov |
Clip Near | The distance to the near clipping plane, in meters. | sensor/camera/clip/near |
Clip Far | The distance to the far clipping plane, in meters. | sensor/camera/clip/far |
Exposure Mode | The exposure mode to use. Has no effect if not using HDRP. DisableAutoExposure avoids visible seams appearing in the image from default auto-exposure. SimulateAutoExposure applies an auto-exposure to the entire fisheye output. DefaultAutoExposure does nothing to the fisheye output. |
sensor/camera/auto_exposure_mode |
Import/Export example
You can import and export a fisheye camera sensor from an XML format similar to the example below:
<sensor name="fisheye" type="fisheyecamera">
<update_rate>5</update_rate>
<topic>/unity/fisheye</topic>
<camera>
<horizontal_fov>3.141592653589793</horizontal_fov>
<image>
<width>1280</width>
<height>720</height>
<format>rgba8</format>
</image>
<clip>
<near>0.1</near>
<far>1000</far>
</clip>
<distortion>
<center>664.5609 371.2716</center>
<k1>-0.013546745328679</k1>
<k2>0.00327781692247613</k2>
<k3>-0.00187067971735455</k3>
<k4>6.13912222927699e-05</k4>
</distortion>
<focal>347.857 346.8416</focal>
</camera>
</sensor>