M6-T8 — Sensor Calibration and Synchronization

Learning objectives

After this lesson, you should be able to:

• Distinguish intrinsic, extrinsic and temporal calibration and say what each one affects.
• Estimate the pixel and position errors caused by a time offset or a lever-arm error.
• Describe how a camera, a camera–IMU pair and a time offset are calibrated.
• Explain rolling-shutter distortion and the difference between hardware and software timestamps.

The problem and the intuition

Imagine tracking a ball by combining two reports: a friend in the stand says where it is, and a radio commentator says where it was a moment ago. If you do not know the commentator is 0.3 s late, you will conclude the ball is behind where your friend says, and you will try to explain the gap by inventing a breeze. A SLAM estimator does the same. An unmodelled time offset between camera and IMU is explained as a gyroscope bias; a wrong lever arm between IMU and camera is explained as an accelerometer bias; a wrong focal length is explained as a scale error. The estimator is not wrong in its logic; it is given the wrong model.

Calibration comes in three kinds. Intrinsic: the properties of one sensor (focal length, principal point, distortion). Extrinsic: where the sensors are relative to each other (a rigid transform). Temporal: when each sensor’s time stamp is relative to the others.

Calibration errors masquerade as other errors. Before blaming drift on the algorithm, check that the sensors’ geometry and clocks are right.

The three calibrations and how they are done

Problem. Determine, with enough accuracy that they are negligible compared with the sensor noise, the intrinsic parameters of each sensor, the rigid transforms between sensors, and the time offsets between their clocks.

Assumptions. The sensors are rigidly mounted (extrinsics are constant); the intrinsics do not change with temperature or focus beyond tolerance; time offsets are constant over the run or slowly varying; the calibration target is known and observed with sufficient excitation.

Intrinsic calibration (camera)

Zhang’s method (2000): observe a planar target with known geometry (a checkerboard or an AprilGrid) from many orientations and minimise the reprojection error over the intrinsics (f_x, f_y, c_x, c_y), the distortion coefficients and the target pose in each image. The quality measure is the final reprojection RMS (a good result is a fraction of a pixel, typically below 0.3 px) and, more usefully, whether it is uniform across the image. Cover the corners of the image, because that is where distortion is largest, and vary the tilt.

Extrinsic calibration

The rigid transform T_CI between camera and IMU, or between camera and lidar, is estimated by moving the rig in front of a target with enough rotation about all axes and acceleration along all axes (Furgale et al., 2013, the Kalibr toolbox). The camera sees the target and provides its pose; the IMU measures rotation rate and acceleration; the optimisation finds T_CI, the biases and the time offset that make them consistent. Excitation matters: motion about a single axis leaves parts of the transform unobservable.

Temporal calibration

Two clocks are rarely the same. The offset t_d in t_IMU = t_cam + t_d can be estimated offline (Kalibr, or by cross-correlating the angular-velocity magnitude from the camera and from the gyroscope, the subject of the fourth lab of this module) or online in the estimator (VINS-Mono, OpenVINS). Better still, avoid the problem with hardware synchronisation: trigger the camera from the same clock that stamps the IMU, or use a shared time source (PPS, PTP) and stamp each measurement at the time of exposure rather than the time the driver delivered it.

Rolling shutter

Most consumer cameras read the sensor row by row, so the top and bottom rows of one image are taken tens of milliseconds apart. A moving camera therefore produces a skewed image: each row has its own pose. The effect is the visual analogue of lidar motion distortion (M5-T2). It can be modelled by letting the pose vary linearly over the readout time and estimating the readout time as a calibration parameter, or avoided with a global-shutter sensor.

K, k_i

intrinsic matrix and distortion coefficients (M6-T1).

T_CI

extrinsic transform from the IMU frame to the camera frame.

t_d

time offset between the camera and IMU clocks.

t_r

rolling-shutter readout time from first to last row.

Effects of calibration errors

A worked example: sizing the errors

1. Time offset. The IMU clock is 10 ms ahead of the camera clock and the robot turns at 2 rad/s. The attitude seen by the IMU at the time of the image differs by ω Δt = 2 × 0.010 = 0.02 rad ≈ 1.15°. A camera with f = 600 px sees a rotation of 0.02 rad as a shift of 600 × 0.02 = 12 px. A 12 px mismatch is far bigger than the sub-pixel residuals a tracker achieves, so the estimator must explain it: it will bias the gyroscope or degrade the features’ weight.
2. Rolling shutter. A sensor with a 30 ms readout and the same 2 rad/s rotation: the bottom row is taken 0.06 rad later than the top row, a 36 px skew over the image if ignored.
3. Lever arm. The IMU is mounted 5 cm from the camera but the extrinsic says 0. During a turn at 2 rad/s the centripetal acceleration at 5 cm is ω² r = 4 × 0.05 = 0.2 m/s². If this is read as accelerometer bias for the 5 s of the turn, it integrates to ½ × 0.2 × 25 = 2.5 m. The same magnitude as a poor accelerometer bias (M6-T6), from a ruler error.
4. Sampling versus offset. At 20 Hz a camera frame has a period of 50 ms; an offset of 10 ms is a fifth of a frame, and not detectable by eye. Because it matters only when the robot rotates, it is often missed in static tests and discovered in the field.
5. Intrinsics. A 1% error in focal length means that a point 20° from the optical axis is predicted 0.2 × (tan 20° × 600 px ≈ 218 px) ≈ 2.2 px off. Small, but systematic and in the same direction for every landmark, so it biases the whole map.

Engineering implications and limitations

Engineering implications

• Calibrate the actual rig, after it is assembled and mounted, and repeat when anything is knocked, re-focused or re-mounted. Keep the result with the dataset.
• Use rich motion for extrinsics. Rotate about all three axes and accelerate along all three; check the estimated parameter covariances rather than only the residual.
• Prefer hardware synchronisation. Trigger the camera from the IMU clock or use PTP/PPS and stamp at mid-exposure. Software stamps (arrival time) include variable USB and driver latency of several to tens of milliseconds.
• Check the offset as part of bring-up. Compare the gyro norm with the camera-derived angular-speed norm and look at their cross-correlation peak. A peak away from zero lag is a clock problem.
• Monitor temperature drift of intrinsics and biases in long deployments; re-estimate online if the estimator supports it.
• Know what is estimated online. Check the estimator’s output for t_d and T_CI: if they wander during a run, the calibration or the excitation is poor.

Limits

Calibration is only as good as its target observations, its model, and the excitation of the calibration motion. Parameters change slowly with temperature, vibration and wear, and some effects (rolling shutter, auto-exposure latency, lens heating) are not captured by the standard model. The standard approach also assumes a constant time offset, which is false when the clocks drift or the driver’s buffering varies.

Common misconceptions

Engineering takeaways

1. Intrinsic, extrinsic and temporal calibration each have distinct signatures when wrong.
2. Time offset error is about f·ω·Δt pixels: 10 ms at 2 rad/s is 12 px with a 600 px focal length.
3. Extrinsic and temporal calibration need rich motion; rotate about all axes and accelerate along all.
4. Hardware synchronisation and stamping at the exposure time remove most timing problems.
5. Check calibration quality by parameter uncertainty, residuals across the image, and the behaviour of the estimated biases.

References and further reading

Primary papers and authoritative documentation.

• Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11).
• Furgale, P., Rehder, J. and Siegwart, R. (2013). Unified temporal and spatial calibration for multi-sensor systems. IEEE/RSJ IROS.
• Furgale, P., Barfoot, T. D. and Sibley, G. (2012). Continuous-time batch estimation using temporal basis functions. IEEE ICRA.
• Qin, T. and Shen, S. (2018). Online temporal calibration for monocular visual-inertial systems. IEEE/RSJ IROS.
• Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press.
• Rehder, J., Nikolic, J., Schneider, T., Hinzmann, T. and Siegwart, R. (2016). Extending Kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes. IEEE ICRA.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

With calibrated sensors and consistent clocks, the final lesson of the module catalogues what still goes wrong in visual SLAM and how to detect and handle it.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T7: Visual-Inertial SLAM

  Compare filter-based (MSCKF) and optimisation-based (VINS-Mono, OKVIS, Kimera-VIO) visual-inertial SLAM.

• Lab
  M6-L4: Diagnose Camera–IMU Synchronization Errors

  Estimate the camera-IMU time offset from angular-velocity signals and see what an uncorrected offset does.