Engineering Guides · Module 18

Implementing watchdogs and health monitoring

Components send heartbeats; a watchdog compares ages with timeouts and decides: stop, restart or warn.

Intermediate
Module 18: Robotics Safety, Reliability & Failure Handling
Written for ROS 2 Jazzy where ROS 2 is used
Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

A component can hang or die without anything else noticing. A watchdog notices the missing heartbeat and triggers a defined reaction, and a lifecycle manager can restart or replace managed nodes in a controlled way.

2. When is this needed?

Designing the supervision of a robot.
After a node hang caused a failure.
Preparing for unattended operation.

3. Selection and design decisions

Facts (read 2026-11-06): ROS 2 managed (lifecycle) nodes have four primary states, unconfigured, inactive, active and finalized, with six transition states and an error-processing path; an external supervisor requests the transitions (design article). Nav2’s lifecycle manager uses this for a controlled startup, shutdown, reset, pause or resume of the navigation stack and allows nodes to be restarted or replaced on-line (its README). The diagnostics packages aggregate component statuses (diagnostic_aggregator). A watchdog is the simple rule that a component whose heartbeat is older than its timeout has failed.

Overview
Component class	Timeout	Reaction
Motor driver link	Very short (100 to 200 ms)	Safe stop
Safety-relevant sensors	Short (0.2 to 1 s)	Safe stop or slow
Localisation	About a second	Slow, then stop
Perception for non-critical functions	Seconds	Degrade and warn
Logging and telemetry	Many seconds	Restart and warn

4. Advantages and disadvantages

Short timeouts react fast and false-trigger under load.
A hardware watchdog survives a hung computer; a software watchdog does not.
Restarting automatically recovers from many faults and can hide a recurring one: count and report restarts.

5. Requirements and constraints

Timeouts chosen from each component’s rate and the stopping distance.
Critical timeouts act on the hardware path, not through a process that may be the one that hung.
Restart counts are logged and escalate.

6. Architecture

components ─ heartbeats ─▶ watchdog ─▶ critical: safe stop / non-critical: restart + warn ─▶ diagnostics aggregator ─▶ operator

7. Implementation

List components and criticality.
Choose a timeout per component.
Implement the watchdog (below) and the reactions.
Add a hardware watchdog for the drive.
Test by killing and hanging each component.

def watchdog(heartbeats, now, timeouts, critical):
    report = []
    for name, to in timeouts.items():
        age = now - heartbeats.get(name, -1e9)
        if age > to:
            report.append((name, round(age, 2), "critical -> safe stop" if name in critical else "non-critical -> restart and warn"))
    return report
timeouts = {"motor_driver": 0.2, "lidar": 0.5, "localisation": 1.0, "logger": 5.0}
beats = {"motor_driver": 9.95, "lidar": 9.2, "localisation": 9.9, "logger": 2.0}
for r in watchdog(beats, 10.0, timeouts, {"motor_driver", "lidar", "localisation"}):
    print("timeout:", r)
print("states of a managed (lifecycle) node: unconfigured, inactive, active, finalized; an error in active leads to error processing")

Printed (checked by the site’s tests):

timeout: ('lidar', 0.8, 'critical -> safe stop')
timeout: ('logger', 8.0, 'non-critical -> restart and warn')
states of a managed (lifecycle) node: unconfigured, inactive, active, finalized; an error in active leads to error processing

Reading it: at t = 10 s the lidar’s last heartbeat is 0.8 s old (timeout 0.5 s) and is critical, so a safe stop is called for; the logger is 8 s old (timeout 5 s) and non-critical, so it is restarted with a warning; the others are within their timeouts. Values are illustrative.

8. Real examples

AMR: drive watchdog in the motor controller firmware stops the wheels if no velocity command arrives for 200 ms.

9. Failure scenarios

A watchdog implemented inside the process that can hang.
Restart loops that hide a bug.
Timeouts longer than the stopping distance allows.

10. Troubleshooting

Kill each component and check the reaction and the time.
Check timing under CPU load.
Review the restart counters.

11. Validation

The watchdog example prints the lines shown (executed by the site tests).
Every component failure leads to the defined reaction within its timeout.

13. Further references

ROS 2 design, Managed nodes (node lifecycle): design.ros2.org, read 2026-11-06.
Nav2 lifecycle manager README (jazzy): nav2_lifecycle_manager, read 2026-11-06.
ROS diagnostics README: ros/diagnostics, read 2026-11-06 (diagnostic_updater publishes, diagnostic_aggregator aggregates, rqt_robot_monitor visualises).