Troubleshooting · Calibration and synchronization
Sensor timestamps are inconsistent
Stamps that disagree with the computer clock, or with each other, break every consumer that interpolates by time. Find which clock each sensor uses and how it differs.
1. Symptoms
- Estimator warnings about old or out-of-order data, or TF extrapolation errors.
- Header stamps far from the current time (seconds, minutes or years in the past).
- Stamps that jump backwards or stop advancing.
- Fusion works for a while, then diverges as the offset grows.
2. Possible causes
- Different clock domains: the sensor stamps with its own clock (power-on time, GPS time) instead of the ROS time.
- Clock drift between the sensor and the computer: the offset grows steadily.
- Stamp convention: the driver stamps at arrival or at scan start/end rather than the sampling instant.
- Simulation time mixed with wall time (
use_sim_timeset on some nodes only). - Network delay and jitter for a networked sensor.
- NTP/PTP stepping the system clock during operation.
3. Measurements to collect
- For each sensor: header stamp minus the receive time, as a time series (mean, spread, trend).
- The ROS time versus the wall time, and whether
use_sim_timeis consistent across nodes. - Out-of-order and missing counts.
- The synchronisation state of the system clock (NTP/PTP).
4. Diagnostic commands
Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).
ros2 topic echo /points --field header.stamp --once # the stamp the driver wrote
date +%s.%N # the computer clock now (Unix time)
ros2 param get /my_node use_sim_time # is it consistent across nodes?
ros2 topic hz /points
ros2 bag record -o stamps /points /imu/data # then analyse with the bag-latency guide's function
timedatectl # is the system clock synchronised? (systemd systems)5. Step-by-step investigation
- Compare the stamp with the clock. If the difference is huge or changes by seconds between runs, the sensor uses another clock domain: configure the driver to use ROS time or the host-synchronised time (PTP, PPS).
- Plot the difference over a long run. A constant value is a fixed offset or latency; a steady slope is drift; steps are clock adjustments.
- Check
use_sim_timeon every node of the launch file; mixed settings cause stamps from two time bases. - Find the stamp convention in the driver documentation or by an event test (a sharp motion visible to two sensors).
- Remove variable delay: use a wired link, hardware timestamping, or measure the jitter and set the synchroniser slop above it.
- Stop clock stepping during operation: configure the time service to slew, not step, or synchronise once before starting.
6. Corrective actions
- Configure the driver for synchronised host time (PTP/PPS) or ROS time.
- Apply a calibrated constant offset where the clock domain cannot be changed, and monitor drift.
- Make
use_sim_timeconsistent across all nodes. - Choose the stamp convention the estimator expects (e.g. exposure time for cameras).
- Rewire or reconfigure the network to bound delay and jitter.
7. Validation after the fix
- Stamp minus receive time is small, stable and without trend over at least 30 minutes.
- No out-of-order or backwards stamps in a 10-minute recording.
- The shared-event test shows the sensors agree within your estimator’s tolerance (use the Sensor Time Synchronization Simulator to decide it).
- Estimator warnings about stale or old data disappear.
References
- See the guide on measuring offsets and latency from ROS bags for the analysis function.
- Sensor Calibration and Synchronization SLAM · Lesson
- Sensor Topics and Message Timing SLAM · Lesson
- Diagnose Sensor Timestamp Problems SLAM · Lab