Camera–LiDAR extrinsic calibration and validation

1. Engineering problem

The extrinsic calibration is the rigid transform between the camera and the LiDAR. Small errors cause visible misalignment: a rotation error moves points by the same pixels at every range, a translation error matters most at close range.

2. When is this needed?

• After mounting or re-mounting sensors.
• When projected points do not line up with the image.
• As a periodic check on a robot in service.

3. Selection and design decisions

Common approaches: target-based (a board or markers seen by both sensors), targetless (maximise agreement between image edges and LiDAR geometry or intensity), and motion-based. Given matched 3D points in both frames the best rigid transform in the least-squares sense is found with the Kabsch algorithm (SVD of the cross-covariance); the points need to be well spread and not all collinear or coplanar for a unique answer.

4. Advantages and disadvantages

• More poses improve accuracy and cost time.
• Initial guess from the CAD model keeps optimisation out of local minima.
• Calibrate at the operating temperature if mounts can flex.

5. Requirements and constraints

• Correspondences are spread through the shared field of view and over several ranges.
• The validation uses data not used in the fit.
• Report error in pixels at specified ranges, not just in metres or degrees.

6. Architecture

matched 3D points (lidar frame A, camera frame B) ─▶ Kabsch/SVD ─▶ R, t ─▶ validate: project held-out points, measure pixel error

7. Implementation

1. Collect matched 3D points in both frames from several board poses at different ranges.
2. Estimate R and t (below) and refine by minimising reprojection error.
3. Compute the residual on held-out poses.
4. Convert the residual to pixels at 5, 10 and 20 m.
5. Store the result with the date, the target and the error figures; publish as a static transform.

import numpy as np
def kabsch(A, B):                                   # find R, t with B ~ R A + t
    ca, cb = A.mean(0), B.mean(0)
    U, _, Vt = np.linalg.svd((A - ca).T @ (B - cb))
    d = np.sign(np.linalg.det(Vt.T @ U.T))
    R = Vt.T @ np.diag([1, 1, d]) @ U.T
    return R, cb - R @ ca
a = np.radians(3.0)
R_true = np.array([[np.cos(a), -np.sin(a), 0], [np.sin(a), np.cos(a), 0], [0, 0, 1]])
t_true = np.array([0.1, -0.05, 0.2])
A = np.array([[2, 0, 0], [3, 1, 0.5], [4, -1, 1.0], [2.5, 0.5, -0.5], [5, 0, 0.2]], dtype=float)
B = A @ R_true.T + t_true
R, t = kabsch(A, B)
err_deg = np.degrees(np.arccos((np.trace(R.T @ R_true) - 1) / 2))
print("rotation error (deg):", round(float(err_deg), 4), " translation error (m):", round(float(np.abs(t - t_true).max()), 6))
f = 600.0
for theta in (0.5, 1.0):
    print(f"extrinsic yaw error {theta} deg: about {f * np.tan(np.radians(theta)):.1f} px at the image centre, at any range")
print("a 0.1 m translation error at 5 m shifts a point by about", round(f * 0.1 / 5.0, 1), "px, at 20 m by", round(f * 0.1 / 20.0, 1), "px")

Printed (checked by the site’s tests):

rotation error (deg): 0.0  translation error (m): 0.0
extrinsic yaw error 0.5 deg: about 5.2 px at the image centre, at any range
extrinsic yaw error 1.0 deg: about 10.5 px at the image centre, at any range
a 0.1 m translation error at 5 m shifts a point by about 12.0 px, at 20 m by 3.0 px

Reading it: the estimator recovers a known 3° rotation and a (0.1, −0.05, 0.2) m translation exactly from noise-free synthetic points. In practice noise limits accuracy; the useful lesson is the sensitivity: with f = 600 px a 0.5° rotation error moves points about 5.2 px and 1° about 10.5 px at the image centre at any range, while a 0.1 m translation error moves a point 12 px at 5 m and 3 px at 20 m. So validate rotation at long range and translation at short range. The values are illustrative.

8. Real examples

• Checkerboard calibration: a board visible in both sensors at 8 poses; fit the transform, then check on 3 other poses.

9. Failure scenarios

• All correspondences near one range and one direction, giving a poorly constrained translation.
• Validation on the same points used for fitting.
• Calibration done indoors and used outdoors where mounts flex with temperature.

10. Troubleshooting

1. If error grows at long range suspect rotation; if it grows at short range suspect translation.
2. If a fit looks perfect and the overlay is wrong, check correspondence quality and time offset.
3. Repeat the calibration to measure repeatability.

11. Validation

• The example prints the values shown (executed by the site tests).
• Held-out reprojection error is below the requirement at 5, 10 and 20 m.
• Repeated calibrations agree within the requirement.

12. Engineering checklist

• Spread of poses and ranges
• Initial guess
• Held-out validation
• Error in pixels at stated ranges
• Time offset checked
• Result stored with date and conditions
• Published as a static transform

13. Further references

• Kabsch algorithm (rigid alignment by SVD) is standard in the literature; the example here is a short implementation checked by the site’s tests.
• ROS 2 sensor_msgs/CameraInfo and Image (Jazzy): ros2/common_interfaces, read 2026-11-05.

Related on FixTheRobot