FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 20

Engineering Guides · Module 20

Remote monitoring and health metrics

Report a few metrics per robot and per fleet, and find the robot that differs from its peers.

  • Intermediate
  • Module 20: Fleet Management & Operational Robotics
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

A fleet generates more data than anyone can read. A handful of metrics per robot, a fleet summary, and automatic detection of outliers show where to look.

2. When is this needed?

  • Setting up a dashboard.
  • Spotting a failing unit early.
  • Reporting to customers.

3. Selection and design decisions

Useful metrics: availability (fraction of scheduled time the robot was usable), task success rate, fault counts by type, battery state, localisation health, mean time between failures. To find outliers, compare each robot with the fleet using a robust statistic: the modified z-score based on the median and the median absolute deviation (MAD), where |z| above 3.5 is a common outlier flag. Diagnostics from each robot can be aggregated with the ROS diagnostics tools (diagnostic_aggregator); remote transport should be tolerant of poor links (buffering and best-effort for telemetry).

MetricUse
AvailabilityContract and planning
Task success rateQuality
Faults by typeWhere to fix
Battery and chargingOperations planning
Software and map versionsConsistency
Link qualityExplains missing data

4. Advantages and disadvantages

  • More metrics give detail and noise.
  • Streaming everything costs bandwidth; summaries plus on-demand logs are cheaper.
  • Alert thresholds tuned too tight create alarm fatigue.

5. Requirements and constraints

  • Telemetry is buffered on the robot through link outages.
  • Each metric has a definition and a unit.
  • Alerts have an owner and a response.

6. Architecture

robots ─ telemetry (buffered) ─▶ aggregator ─▶ fleet metrics, outlier detection ─▶ dashboard and alerts

7. Implementation

  1. Define the metrics and their definitions.
  2. Collect them per robot with timestamps.
  3. Compute fleet summaries and outliers (below).
  4. Set alerts with owners.
  5. Review weekly and refine.
import statistics as st
robots = {"R1": {"uptime": 0.97, "tasks": 120, "ok": 118, "faults": 1}, "R2": {"uptime": 0.96, "tasks": 118, "ok": 115, "faults": 2}, "R3": {"uptime": 0.95, "tasks": 121, "ok": 119, "faults": 2},
          "R4": {"uptime": 0.71, "tasks": 80, "ok": 60, "faults": 9}, "R5": {"uptime": 0.97, "tasks": 119, "ok": 117, "faults": 1}}
up = [r["uptime"] for r in robots.values()]
med = st.median(up); mad = st.median([abs(u - med) for u in up])
print(f"fleet availability (mean uptime) {sum(up) / len(up) * 100:.1f} %, task success {sum(r['ok'] for r in robots.values()) / sum(r['tasks'] for r in robots.values()) * 100:.1f} %")
for name, r in robots.items():
    z = 0.6745 * (r["uptime"] - med) / mad if mad else 0.0
    flag = "OUTLIER" if abs(z) > 3.5 else "normal"
    print(f"{name}: uptime {r['uptime'] * 100:.0f} %, success {r['ok'] / r['tasks'] * 100:.0f} %, faults {r['faults']}, robust z {z:6.1f} {flag}")

Printed (checked by the site’s tests):

fleet availability (mean uptime) 91.2 %, task success 94.8 %
R1: uptime 97 %, success 98 %, faults 1, robust z    0.7 normal
R2: uptime 96 %, success 97 %, faults 2, robust z    0.0 normal
R3: uptime 95 %, success 98 %, faults 2, robust z   -0.7 normal
R4: uptime 71 %, success 75 %, faults 9, robust z  -16.9 OUTLIER
R5: uptime 97 %, success 98 %, faults 1, robust z    0.7 normal

Reading it: the fleet has 91.2 % mean availability and a 94.8 % task success rate; robot R4 at 71 % uptime, 75 % success and 9 faults has a robust z-score of −16.9 and is flagged as an outlier while the others are normal. The numbers are illustrative.

8. Real examples

  • Weekly review: one robot’s repeated localisation faults traced to a damaged sensor mount.

9. Failure scenarios

  • Averages hiding a failing robot.
  • Alerts nobody owns.
  • Telemetry lost whenever the link drops.

10. Troubleshooting

  1. Compare the outlier robot with its peers on hardware, software and site.
  2. Check link quality for missing data.
  3. Check metric definitions are the same on all robots.

11. Validation

  • The summary prints the lines shown (executed by the site tests).
  • Outliers are flagged within a day of becoming outliers.

12. Engineering checklist

13. Further references

  • ROS diagnostics README: ros/diagnostics, read 2026-11-06. The modified z-score is a standard robust statistic (Iglewicz and Hoaglin); not read here.

Related on FixTheRobot