FixTheRobot Explore Robotics / Systems Engineering / Troubleshooting

Troubleshooting · Calibration and synchronization

Camera data is delayed

Latency is the sum of exposure, readout, transfer, driver, transport and processing. Measure each stage by comparing stamps.

  • Related modules: 07
  • ROS 2 Jazzy

1. Symptoms

  • The camera view lags reality; detections trail the moving object.
  • Fusion with other sensors is misaligned only when moving.
  • The frame rate is right but the delay is large.

2. Possible causes

  • Long exposure in low light and a slow readout.
  • Transfer over a congested USB or Ethernet link.
  • Driver buffering or software conversion (for example colour conversion on the CPU).
  • Large uncompressed images through DDS, or a reliable QoS retrying lost packets.
  • Subscribers that are slower than the stream, building a queue.
  • Stamping at arrival so the delay is hidden in the stamp.

3. Measurements to collect

  • now − header.stamp at the subscriber for each frame.
  • Frame rate at the driver output and at the consumer.
  • CPU load and network utilisation.
  • Exposure time.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

ros2 topic delay /camera/image_raw
ros2 topic hz /camera/image_raw
ros2 topic bw /camera/image_raw
ros2 topic info /camera/image_raw --verbose

5. Step-by-step investigation

  1. Measure the delay from the header stamp to arrival at the subscriber.
  2. Check that the stamp is the exposure time; if it is the arrival time the true delay is larger.
  3. Compare the rate at the driver and at the consumer; a drop shows a queue.
  4. Reduce the data (resolution, compression) and see whether the delay falls.
  5. Check QoS (best-effort with a small queue for streaming video).
  6. Check CPU for conversions and slow subscribers.

6. Corrective actions

  • Cap exposure and raise gain or light.
  • Use a faster interface or reduce resolution and rate.
  • Use hardware colour conversion or a compressed transport where acceptable.
  • Use best-effort QoS with a short queue for live video.
  • Compensate the known delay in the estimator or use interpolation.

7. Validation after the fix

  • End-to-end delay is below the requirement at the highest load.
  • Delay stays stable over an hour.

References