FixTheRobot Explore Robotics / Systems Engineering / Troubleshooting

Troubleshooting · Compute and ROS 2

Messages are dropped

The subscriber sees fewer messages than the publisher sent. Find where they disappear: QoS, queue, CPU, network or the recorder itself.

  • Related modules: 17
  • ROS 2 Jazzy

1. Symptoms

  • A subscriber’s rate is lower than the publisher’s.
  • Estimator or perception output has gaps.
  • A bag has fewer messages than expected.
  • Drops only under load or only on a networked machine.

2. Possible causes

  • QoS incompatibility: reliability or durability mismatch between publisher and subscriber (best-effort publisher with a reliable subscriber, for example, will not connect).
  • Queue depth too small for bursts: old messages are overwritten.
  • Subscriber too slow: callbacks take longer than the message period.
  • Network limits for large messages (bandwidth, fragmentation, Wi-Fi).
  • Best-effort delivery over a lossy link.
  • CPU or memory pressure or a blocked executor.
  • Recorder drops: the bag writer cannot keep up.

3. Measurements to collect

  • Publisher rate versus subscriber rate for the same topic.
  • QoS profiles of publisher and subscriber.
  • Bandwidth of the topic and the link capacity.
  • Callback duration of the slow subscriber.
  • CPU, memory and network counters during the drops.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

ros2 topic info -v /points        # publishers, subscribers and their QoS profiles
ros2 topic hz /points             # rate seen by this subscriber
ros2 topic bw /points             # bandwidth seen by this subscriber
ros2 doctor                       # basic checks of the ROS 2 setup
ros2 daemon stop && ros2 daemon start    # if discovery looks stale
htop                              # CPU load while it happens

5. Step-by-step investigation

  1. Compare rates at the publisher’s own node (its internal counter or log) and with ros2 topic hz at the subscriber.
  2. Compare QoS with ros2 topic info -v. A reliability or durability mismatch can prevent data flowing at all, or cause a different behaviour than expected.
  3. Check queue depth against burst size: a depth of 1 at a bursty source loses messages whenever the subscriber is briefly busy.
  4. Time the callbacks. If one callback takes longer than the period, move work out of the callback or use a multi-threaded executor with a separate callback group.
  5. Check the link. Compare the topic bandwidth with the link capacity (see the throughput guide); try the same topic on the same machine to separate network from CPU.
  6. Rule out the recorder: check CPU and disk write speed while recording; record fewer topics to see whether drops disappear.

6. Corrective actions

  • Match QoS to the data: sensor streams usually best-effort with a small queue, state topics reliable.
  • Increase queue depth for bursty sources (and understand it increases latency).
  • Make callbacks fast; do heavy work in a separate thread or node.
  • Reduce message size or rate; compress or crop images; downsample point clouds before sending over the network.
  • Run producer and consumer on the same machine, or use a wired link.
  • Give the recorder faster storage or fewer topics.

7. Validation after the fix

  • Publisher and subscriber message counts agree over a 10-minute run.
  • Rates are stable with no gaps in stamps (see the bag-latency guide).
  • Drops do not reappear at maximum load.
  • The fix holds after a restart and on the networked machine.

References

  • ROS 2 QoS documentation for your distribution (Jazzy): reliability, durability, history and depth. Not fetched in the build environment.