Measuring offsets and latency from ROS bags

1. Engineering problem

A fused estimator misbehaves only in some runs, and the timing is the suspect. Before touching parameters, measure: how old is each sensor’s data on arrival, are messages missing or reordered, and is the sensor’s clock consistent with the computer’s? A recorded bag lets you do this offline, repeatably.

2. When is this needed?

• Zig-zag or jumping estimates in LiDAR-inertial or visual-inertial odometry.
• Camera and LiDAR detections disagree only while the robot moves.
• After changing a driver, a cable, a network setting or a time-sync configuration.

3. Selection and design decisions

Two clocks are in every bag message: stamp (header.stamp, written by the driver) and receive time (the bag’s recorded timestamp, from the recorder’s clock). They answer different questions.

Latency alone cannot separate clock offset from delay. If the sensor is on the same machine clock, latency should be small and positive; a constant ≈ 0 with a large negative or positive bias points at a clock-domain problem.

4. Advantages and disadvantages

• Offline bag analysis: repeatable, no load on the robot, but only sees what was recorded and is blind to what the recorder itself dropped.
• Live ros2 topic hz / bw: instant and good for rate and bandwidth, but gives no stamp-versus-receive information.
• Hardware timing (logic analyser, PPS capture): ground truth for clock offsets, needs equipment and access to signals.

5. Requirements and constraints

• The recorder must keep up: a bag that dropped messages hides the very drops you are chasing. Record with enough buffer and check ros2 bag info message counts against expected rates.
• The bag’s receive timestamp has its own latency (subscription queue, serialisation). Treat latency numbers as upper bounds and compare between runs and sensors.
• Use the same QoS as the real consumer. A best-effort subscription can drop messages that a reliable one would keep.
• Message types without a header (some vendor messages) have no stamp to analyse.

6. Architecture

The recorder subscribes to the same topics as the estimator, stamps each message with its own receive time, and writes it to the bag:

sensor ──(driver stamps header.stamp)──▶ topic ──▶ recorder ──(bag timestamp = receive time)──▶ bag
                                           └────▶ estimator

7. Implementation

Record

Not executed by the site’s automated tests (they need a ROS 2 installation).

ros2 bag info my_bag                                  # message counts per topic, duration, storage
ros2 bag record -o my_bag /imu /points /camera/image_raw
ros2 topic hz /points                                 # live rate
ros2 topic bw /points                                 # live bandwidth

The analysis function (tested)

def analyze(stamps, recv, period):
    """stamps: header.stamp of each message in RECEIVED order (s); recv: when it was recorded (s); period: nominal 1/rate (s)."""
    latency = [r - s for s, r in zip(stamps, recv)]
    out_of_order = sum(1 for a, b in zip(stamps, stamps[1:]) if b <= a)       # stamp did not increase
    srt = sorted(stamps)
    missing = sum(max(0, round((b - a) / period) - 1) for a, b in zip(srt, srt[1:]))   # whole periods skipped
    return {"n": len(stamps), "mean_latency_ms": round(1000 * sum(latency) / len(latency), 1),
            "max_latency_ms": round(1000 * max(latency), 1), "out_of_order": out_of_order, "missing": missing}

# a 10 Hz camera: message 0.4 was dropped, 0.6 arrived late and after 0.7
stamps = [0.0, 0.1, 0.2, 0.3, 0.5, 0.7, 0.6, 0.8, 0.9]
recv   = [0.05, 0.15, 0.25, 0.35, 0.55, 0.75, 0.751, 0.85, 0.95]
print(analyze(stamps, recv, 0.1))

Printed result for the synthetic 10 Hz camera (checked by the site’s tests):

{'n': 9, 'mean_latency_ms': 61.2, 'max_latency_ms': 151.0, 'out_of_order': 1, 'missing': 1}

Reading a real bag (not executed by the tests)

import rosbag2_py
from rclpy.serialization import deserialize_message
from rosidl_runtime_py.utilities import get_message

reader = rosbag2_py.SequentialReader()
# storage_id is "sqlite3" or "mcap": use the value that "ros2 bag info" reports
reader.open(rosbag2_py.StorageOptions(uri="my_bag", storage_id="sqlite3"), rosbag2_py.ConverterOptions("", ""))
types = {t.name: get_message(t.type) for t in reader.get_all_topics_and_types()}
stamps, recv = [], []
while reader.has_next():
    topic, data, t_ns = reader.read_next()
    if topic == "/points":
        msg = deserialize_message(data, types[topic])
        stamps.append(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9)
        recv.append(t_ns * 1e-9)
# then: analyze(stamps, recv, 1 / 10.0)

8. Real examples

1. Constant 50 ms latency on the camera, 10 ms on the IMU. A fixed pipeline delay. If stamps are the exposure time it does not corrupt the geometry; a delayed-measurement filter must handle it. Check with the Sensor Time Synchronization Simulator.
2. Latency starts at 5 ms and grows by 1 ms per minute. Sensor clock drift against the computer clock: use PTP or PPS, not a one-time offset.
3. Latency near zero but LiDAR–IMU zig-zag remains. Both are stamped consistently relative to the receive clock, yet the stamp convention differs (scan start vs end) or an extrinsic is wrong: compare against an event, not against the receive time.

9. Failure scenarios

• Analysing a bag that dropped messages and concluding the sensor is fine.
• Mistaking recorder-side queueing for sensor latency.
• Comparing stamps of two sensors that use different clock domains without a common event.
• Ignoring that a LiDAR stamp can mean scan start, scan end or mid-scan depending on the driver.

10. Troubleshooting

1. ros2 bag info: compare message count with duration × rate for each topic.
2. Run the analysis on each sensor: mean and maximum latency, reordering, missing count.
3. Constant latency → pipeline delay or constant offset. Growing → drift or backlog. Negative → sensor clock ahead.
4. For relative offsets, induce a shared event (a sharp rotation seen by IMU and camera, or a LiDAR-visible edge crossing the camera view) and compare the stamp of the event in each stream.
5. Fix the cause (time sync, driver stamping option, QoS or CPU load), record again, repeat the measurement.

11. Validation

• After the fix, latency for each sensor is stable (small spread) over a 10-minute recording.
• Zero out-of-order messages and no missing messages at nominal rates.
• The shared-event stamp difference between sensors is below the tolerance of your estimator (decide it from the simulator and the estimator’s documentation).
• The analysis function’s output on the synthetic example matches the values printed above (executed by the site tests).

12. Engineering checklist

• Bag message counts match expected rates
• Receive − stamp measured per sensor, with mean, spread and trend
• Out-of-order and missing counts recorded
• Stamp convention of each sensor known (start, end, exposure)
• Same QoS as the consumer when recording
• A common-event test done for every sensor pair that is fused
• Before and after numbers saved with the fix

13. Further references

• ROS 2 bag tools: ros2 bag info, ros2 bag record and the rosbag2_py Python API, ROS 2 Jazzy documentation. Not fetched in the build environment; verify argument names in your installation (ros2 bag record --help).

Related on FixTheRobot

• Sensor Calibration and Synchronization SLAM · Lesson
• Sensor Topics and Message Timing SLAM · Lesson
• Diagnose Sensor Timestamp Problems SLAM · Lab
• Diagnose Camera–IMU Synchronization Errors SLAM · Lab
• Coordinate Alignment and Timestamps Perception · Lesson