Writing a reliable hardware driver

1. Engineering problem

Cables loosen, devices reset and ports vanish. A driver that assumes perfect hardware crashes or hangs. A reliable driver treats failure as a normal state it can detect, report and recover from.

2. When is this needed?

• Writing a driver for a new sensor or actuator.
• When a device occasionally disappears and the node must recover.
• When reviewing a driver before field use.

3. Selection and design decisions

A reliable driver has these parts: an open and configure step; a read loop with a timeout; a watchdog that notices silence; reconnect with back-off; diagnostics published to ROS 2 (for example with the diagnostic_updater package); and parameters instead of constants.

4. Advantages and disadvantages

• Fast retry recovers sooner and can hammer a failing device; back-off is gentler and slower to recover.
• Crash-and-restart by a supervisor is simple; in-driver recovery avoids losing state.
• More diagnostics cost a little CPU and make field debugging possible.

5. Requirements and constraints

• The read loop never blocks without a timeout.
• An actuator driver sets a safe output when commands stop arriving.
• Reconnect attempts are rate limited.

6. Architecture

open ─▶ read loop (timeout) ──data──▶ publish
                 │ silence > timeout
                 ▼
           report error ─▶ wait (back-off) ─▶ reopen

7. Implementation

1. Define the failure conditions: no data, bad data, device disappeared.
2. Add a timeout to every read and a watchdog on message arrival.
3. Implement reopen with capped exponential back-off (below).
4. Publish diagnostics and log state changes once.
5. Test by pulling the cable, resetting the device and filling the CPU.

def run(events, timeout=1.0, backoff=(0.5, 1.0, 2.0)):
    last, tries, log = 0.0, 0, []
    for t, ok in events:
        if ok:
            last, tries = t, 0
            continue
        if t - last > timeout:
            delay = backoff[min(tries, len(backoff) - 1)]
            log.append(f"t={t:.1f}s no data for {t - last:.1f}s: reopen after {delay}s")
            tries += 1
    return log
print("\n".join(run([(0.5, True), (1.0, True), (2.5, False), (3.0, False), (4.5, False)])))

Printed (checked by the site’s tests):

t=2.5s no data for 1.5s: reopen after 0.5s
t=3.0s no data for 2.0s: reopen after 1.0s
t=4.5s no data for 3.5s: reopen after 2.0s

Reading it: data arrives until 1.0 s, then stops. At 2.5 s the silence (1.5 s) exceeds the 1.0 s timeout and the first reopen delay is 0.5 s; later checks double it to 1.0 s and 2.0 s, capped at the last value. Times and delays are illustrative.

8. Real examples

• Motor driver link: on loss of commands for 200 ms the driver commands zero velocity and reports an error.

9. Failure scenarios

• A read call that blocks forever after the cable is pulled.
• Reopen attempts in a tight loop that starve the CPU.
• A driver that logs the same error 1000 times per second.

10. Troubleshooting

1. If the node hangs, find the blocking call.
2. If it will not reconnect, check that the port is closed before reopening and that the device name is stable.
3. If CPU is high during failure, check retry rate and logging.

11. Validation

• The example prints the lines shown (executed by the site tests).
• Pulling and replugging the device 20 times recovers each time within the expected delay.
• Diagnostics show the true state.

12. Engineering checklist

• Read timeout
• Watchdog
• Back-off with cap
• Diagnostics published
• Safe actuator outputs
• Parameters
• Failure injection tested

13. Further references

• ROS 2 diagnostics documentation: docs.ros.org (diagnostic_updater package).

Related on FixTheRobot

• Loop Closure SLAM · Lesson