Nodes stop responding

1. Symptoms

• A node stops publishing and its services time out.
• CPU is at zero for the node.
• It recovers after a restart and fails again later.

2. Possible causes

• A synchronous call inside a mutually exclusive callback group waiting on a response that needs the same group.
• A blocking call (I/O, sleep, lock) in a callback.
• A single-threaded executor starved by a long callback.
• Memory exhaustion or the process killed by the OS.
• A crashed node with no supervisor to restart it.

3. Measurements to collect

• Thread states of the process (stack dump).
• The callback group configuration.
• dmesg for out-of-memory kills.
• Node liveness from ros2 node list and the diagnostics.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

ros2 node list
ros2 node info /my_node
gdb -p <PID> -batch -ex "thread apply all bt"
dmesg | grep -i -E "killed process|out of memory"

5. Step-by-step investigation

1. Check whether the process exists and its memory.
2. Dump the thread stacks to see where it is blocked.
3. Review callback groups for the synchronous-call deadlock pattern.
4. Check for blocking calls in callbacks.
5. Check the OS logs for out-of-memory kills.

6. Corrective actions

• Use asynchronous calls or a separate callback group for responses (see the Jazzy callback-group guidance).
• Move blocking work to its own thread.
• Fix the memory growth.
• Add a supervisor or lifecycle management to restart and report.

7. Validation after the fix

• Long-run test without hangs, and a watchdog that detects a blocked node.

References

• ROS 2 Jazzy, About executors: docs.ros.org, and Using callback groups: docs.ros.org, read 2026-11-06.
• See the QoS and executors guide.