Camera Calibration

Distortion Types of RGB Cameras

Lens cameras can cause distortion. So we should calibrate for distortions. Different types of distortions include:

• radial

• tangential

Camera Calibration Parameters

Intrinsic parameters are those things that are inherent to the camera itself, namely the focal length and optical center as well as the distortion coefficients. These properties remain the same no matter how the camera is positioned within the world. The extrinsic parameters of the calibration process describe how the camera's reference frame is oriented within the world reference frame. Computing the mapping between image points and world points depends on both the intrinsic and extrinsic parameters.

Camera Calibration Methods

Camera calibration using the calibration pattern

To calibrate your camera, you need to measure how 3D points in the world get mapped down onto the 2D image plane in the camera. In reality, that's a rather complicated process. The process is as below:

• The first step is to choose a calibration pattern.

• Take pictures of the calibration pattern.

• Find corners.

• Use cv2.findChessboardCorners() to find corners in chessboard images and aggregate arrays of image points (2D image plane points) and object points (3D world points).

• Use the OpenCV function cv2.calibrateCamera() to compute the calibration matrices and distortion coefficients.

• Use cv2.undistort() to undistort a test image.

In general, you need to calibrate using many images of the test pattern in order to get a good sampling of points across the image plane. Given object points, image points, and the (x, y) shape of the image, you're ready to perform the calibration. Check out this tutorial for more detail on camera calibration with OpenCV. You'll call the cv2.calibrateCamera() function like this:

ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, (img.shape[1], img.shape[0]), None, None)

The important outputs to focus on here are mtx and dist, which contain the intrinsic camera matrix and distortion coefficients, respectively. The intrinsic camera matrix is typically written like this:

$$\text { camera matrix }=\left[\begin{array}{ccc} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{array}\right]$$

In the above matrix, $f_{x}$ and $f_{y}$ correspond to the focal length (and they are equal when pixels are square), while $c_{x}$ and $c_{y}$ correspond to the camera's optical center in the $(x,y)$ plane. There are five distortion coefficients in dist and they are given in this order:

$$\text { Distortion Coefficients }=\left(\mathbf{k}_1, \mathbf{k}_2, \mathbf{p}_1, \mathbf{p}_2, \mathbf{k}_3\right)$$

where the $\mathbf{k}$'s are the radial distortion coefficients and the $\mathbf{p}$'s are the tangential distortion coefficients. To correct an image for radial and tangential distortion, you'll use the cv2.undistort() function to apply to the following equations. The equation for radial distortion is as follows:

$$\begin{aligned} \mathrm{x}_{\text {corrected }} & =\mathrm{x}\left(1+\mathrm{k}_1 \mathrm{r}^2+\mathrm{k}_2 \mathrm{r}^4+\mathrm{k}_3 \mathrm{r}^6\right) \\ \mathrm{y}_{\text {corrected }} & =\mathrm{y}\left(1+\mathrm{k}_1 \mathrm{r}^2+\mathrm{k}_2 \mathrm{r}^4+\mathrm{k}_3 \mathrm{r}^6\right) \end{aligned}$$

And for the tangential distortion, the equation is:

$$\begin{aligned} & x_{\text {corrected }}=x+\left(2 \mathbf{p}_1 \mathbf{x y}+\mathbf{p}_2\left(\mathbf{r}^2+2 \mathrm{x}^2\right)\right) \\ & {y}_{\text {corrected }}=\mathbf{y}+\left(\mathbf{p}_1\left(\mathbf{r}^2+2 \mathbf{y}^2\right)+2 \mathbf{p}_2 \mathbf{x y}\right) \end{aligned}$$

To apply undistortion to an image, you can call cv2.undistort() like this:

undist = cv2.undistort(img, mtx, dist, None, mtx)

Pinhole Camera Model for Extrinsic Calibration

The relationship between the position of a point in the 2D pixel coordinates and the position of a corresponding point in the 3D world coordinates, is defined by the pinhole camera model:

$$z_c\left[\begin{array}{c} u \\ v \\ 1 \end{array}\right]=K[R T]\left[\begin{array}{c} x_w \\ y_w \\ z_w \\ 1 \end{array}\right]$$

where:

• $\boldsymbol{K}$ represents the intrinsic matrix derived in the previous section.

• $\boldsymbol{R}$ and $\boldsymbol{T}$ are the extrinsic parameters which denote the coordinate system transformations from 3D world coordinates to 3D camera coordinates. Equivalently, the extrinsic parameters define the position of the camera center and the camera's heading in world coordinates.

• $\boldsymbol{T}$ is the position of the origin of the world coordinate system expressed in coordinates of the camera-centered coordinate system. While, the position of the camera is $\boldsymbol{C}= -\boldsymbol{R}^T\boldsymbol{T}$ where $\boldsymbol{R}^T$ is the transpose of matrix $\boldsymbol{R}$.

Since the extrinsic parameters determine the 3D pose of the respective camera in world coordinates, they can also be used for the process of image registration.

Image registration refers to transforming the data frame from one camera to match the data frame from the other camera, pixel by pixel. This is absolutely essential for the creation of an accurate point cloud.

RGBD Calibration in ROS

An RGB-D camera can be calibrated using ROS image_pipeline, which contains a suite of packages that provide a wide variety of image manipulation tools. One of those tools is the camera_calibration package that allows easy calibration of monocular, stereo, and RGB-D cameras. Check out this tutorial for more information about the tools that ROS provides for calibration. The calibration process will allow you to combine the RGB camera data with the accompanying depth camera data to generate 3D point clouds.