P8-T3 — Coordinate Alignment and Timestamps

Learning objectives

After this lesson, you should be able to:

• Express a point from one sensor frame in another with a rigid transform and name the extrinsic parameters.
• Compute the position error caused by a rotation error, a translation error and a time offset.
• Explain timestamps, clock synchronisation and interpolation, and the distortion caused by motion during a scan.
• Diagnose a misalignment from the pattern of disagreement between sensors.

The idea: agree on where and when

Every sensor reports in its own frame: the camera’s origin is at the lens with the z axis forward and y down; the lidar’s origin is at its mirror; the robot’s frame is at its centre, on the ground. To compare their points you must move them into one frame. And every reading belongs to a moment: a lidar sweep takes 100 ms, a camera frame is taken at an instant, a radar scan another. To combine them you must align them in time as well.

Frames and extrinsics

The extrinsic calibration is the rigid transform between two sensor frames: a rotation (three angles) and a translation (three distances). It is found once, by measuring or by optimisation (Lab P8-L3, stage 2), and assumed to stay constant, which is only true if the mount is rigid. The intrinsic calibration belongs to one sensor: a camera’s focal length, principal point and distortion (Lesson P3-T2). Fusion needs both: intrinsics to turn pixels into rays, extrinsics to put the rays and the lidar points in one frame.

Timestamps and clocks

Each measurement carries a timestamp. Three things go wrong. The timestamp may be applied when the data arrive at the computer instead of when they were captured, which adds the transfer and processing delay. The clocks of two devices may drift or start from different origins. And a sensor may measure over an interval (a lidar sweep) and report a single time. The remedies are hardware triggering (one signal starts both sensors), clock synchronisation protocols (such as PTP over a network or GPS time), stamping at capture, and compensating for the interval.

Interpolation and motion distortion

Two sensors rarely measure at the same instant. To compare them, bring one to the time of the other. For a quantity that changes smoothly, such as a pose, linear interpolation between the two nearest samples is enough. A lidar sweep, taken while the robot moves, is distorted: points at the end of the sweep were measured from a different pose than points at the start. The fix is to give each point its own time and move it into a common frame with the pose interpolated at that time (de-skewing).

Reading the symptom

The pattern of disagreement tells the cause. A constant offset between the sensors for every object is a translation error. A disagreement that grows with distance from the robot, perpendicular to the line of sight, is a rotation error. A disagreement that is zero for a stationary object and grows with speed, along the direction of motion, is a time offset. A disagreement that grows when the robot turns is a time offset combined with the robot’s rotation. Mixed faults need all of these checks.

The mathematics: transforms and error growth

A point p_A in the frame of sensor A is expressed in the frame of sensor B by the rigid transform

R

a rotation matrix (3 × 3, or 2 × 2 on the ground plane) with R Rᵀ = I and det R = 1: the orientation of frame A as seen from B.

t

the translation: the position of the origin of A expressed in B.

p_A, p_B

the same physical point, in the coordinates of A and of B.

The inverse is p_A = Rᵀ(p_B − t). Chaining frames multiplies the transforms, so the order matters, and the errors of each link add.

Rotation error. If the stored rotation is wrong by a small angle δ, a point at range r is displaced sideways by

Translation error. A translation error Δt displaces every point by Δt, independent of range.

Time offset. If the two sensors’ data are τ apart and the object moves at velocity v, the position error along the motion is

If the robot itself turns at rate ω, an object at range r appears displaced sideways by ω τ r, so a time offset looks like a rotation error whenever the robot turns.

Interpolation. A pose known at times t₀ and t₁ as x₀ and x₁ has, at a time t between them, the linear estimate

(Rotations need spherical interpolation of the orientation, but for the small angles between neighbouring samples, linear interpolation of the angle is a good approximation.)

Worked examples: how big the errors get

Rotation. The camera’s stored yaw is wrong by 0.5° = 0.00873 rad. At 5 m the lateral error is 0.044 m, which is barely noticeable. At 20 m it is 0.175 m, about a third of a person’s width (0.5 m); at 50 m it is 0.44 m, almost a whole person width. The error that nobody sees on the bench matters at a distance. In the lab’s fault of 3° the error is 0.52 m at 10 m.

Translation. A camera that has slipped 0.1 m on its mount is wrong by 0.1 m at every range. It matters most at short range, where the error is a large fraction of the distance, and for manipulation, where centimetres count.

Time. A camera frame is stamped 30 ms late and the robot passes a pedestrian crossing at 1.5 m/s: error = 1.5 × 0.03 = 0.045 m, small. Two vehicles approaching at 15 m/s each give a relative speed of 30 m/s, and the same 30 ms offset puts the positions 0.9 m apart. For a single vehicle at 15 m/s the error is 0.45 m. A 50 ms offset on a robot turning at 0.5 rad/s, looking at an object at 20 m: lateral error = 0.5 × 0.05 × 20 = 0.5 m, as large as a person’s width, and indistinguishable from a rotation error unless you check against the turn rate.

Motion distortion. A lidar sweep takes 0.1 s and the robot drives at 10 m/s. The robot moves 1 m during the sweep, so the first and the last points of one scan are 1 m apart in the world even if they hit the same wall. A de-skewing step that uses the pose at each point’s time removes this.

Interpolation. A lidar scan is stamped at t = 1.037 s. The robot pose is known at 1.0 s with x = 10.0 m and at 1.1 s with x = 11.5 m. The scan’s pose is x = 10.0 + (1.037 − 1.0) / (1.1 − 1.0) × (11.5 − 10.0) = 10.0 + 0.37 × 1.5 = 10.555 m. Taking the pose at the nearest stamp, 1.0 s, would place the scan 0.555 m off.

A diagnostic procedure

1. Check at rest. With the robot and the scene still, compare the sensors on distinct objects. Disagreement that remains is calibration; it should be zero within noise.
2. Plot the residual against range. A constant residual is a translation; one that grows linearly with range, perpendicular to the line of sight, is a rotation (Lab P8-L3, stage 1).
3. Repeat with motion. If the residual appears only when things move, and lies along the motion, it is a time offset. Find it by sliding one sequence against the other and minimising the disagreement (Lab P8-L3, stage 3).
4. Estimate and re-check. Compute the correction from matched points, apply it, and verify on data that were not used to estimate it.

Engineering implications and limitations

Treat calibration as a part that can fail. Mounts loosen, bumpers get knocked, temperature shifts lenses. Add a check that runs on logged data, comparing the sensors on distinct objects, and raise an alarm when the disagreement grows (Lesson P8-T5).

Calibrate with data spread over range and direction. A rotation is confounded with a translation if all targets are at one distance. Use points over a range of distances and bearings, as the matched points of Lab P8-L3 are.

Stamp at capture. Use hardware time stamps from the sensor, or a hardware trigger. A software timestamp on arrival includes a variable delay that can be tens of milliseconds.

Synchronise the clocks. Use PTP or GPS time on every device, and monitor the offsets. Record the offset between clocks with the data, so that a log can be corrected afterwards.

Interpolate, do not snap. Bring data to a common time by interpolation, and say in the interface which time a message refers to.

Do not fuse until the check passes. A fusion that runs on misaligned data produces confident wrong results. Make “sensors aligned” a condition that the system verifies, not an assumption.

Keep frame conventions explicit. Axes (right-down-forward versus forward-left-up), units, and the direction of the transform (A to B or B to A) cause more bugs than the mathematics. Name transforms T_B_from_A and keep a single table of frames for the whole system.

Common misconceptions

Engineering takeaways

1. A point is moved between sensor frames by p_B = R p_A + t; the extrinsics are R and t, found by calibration and assumed rigid.
2. Rotation error gives a lateral error r·δ (44 cm at 50 m for 0.5°); translation error is constant; time offset gives v·τ.
3. Timestamps must be applied at capture, clocks synchronised, and data interpolated to a common time; a moving lidar sweep must be de-skewed.
4. The pattern of disagreement identifies the fault: constant means translation, growing with range means rotation, speed-dependent means time.
5. Verify alignment as part of operation: fusing misaligned sensors gives confident wrong results.

References and further reading

Textbooks and papers for the methods named. The example speeds, offsets and angles are illustrative.

• Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: camera models and rigid transforms.
• Furgale, P., Rehder, J. and Siegwart, R. (2013). Unified temporal and spatial calibration for multi-sensor systems. IROS.
• Zhang, J. and Singh, S. (2014). LOAM: Lidar Odometry and Mapping in Real-time. Robotics: Science and Systems: motion distortion correction of lidar sweeps.
• Umeyama, S. (1991). Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence: the closed-form alignment used in stage 2 of Lab P8-L3.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: where in the pipeline to combine the sensors: early, late and object-level fusion.

Related content

Nothing below is required to finish this lesson.

• Before
  P8-T2: Complementary Strengths and Weaknesses of Cameras and LiDAR

  Compare what cameras and LiDAR measure well and badly.

• Lab
  P8-L3: Diagnose Incorrect Coordinate Transforms

  Find a wrong extrinsic or time offset from how fused data disagree.