FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 20

Engineering Guides · Module 20

Diagnosing fleet-wide failures

Count incidents per robot in each group. The factor that explains the failures across groups is the cause.

  • Intermediate
  • Module 20: Fleet Management & Operational Robotics
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

When many robots fail, the question is what they have in common. Raw incident counts mislead because groups differ in size; incidents per robot in each group reveal the factor.

2. When is this needed?

  • A rise in incidents across the fleet.
  • After a rollout.
  • Before deciding whether to roll back.

3. Selection and design decisions

Group the incidents by candidate factors: software version, site, hardware batch, map version, time of day, network. For each group compute incidents per robot (incident count divided by the number of robots in that group). A factor that separates high from low rates across the other factors is the likely cause: a version that is bad at every site is more convincing than a site that is bad for every version. For confidence use a statistical test appropriate to the counts.

FactorTest
Software versionRate per robot per version, across sites
SiteRate per robot per site, across versions
Hardware batchRate by batch
Map versionRate by map version
TimeRate by hour or day (shifts, weather, network load)

4. Advantages and disadvantages

  • Many factors raise the chance of a spurious pattern; check with a second grouping.
  • Rolling back as a test is fast and informative when the version is suspected.
  • Small groups give noisy rates.

5. Requirements and constraints

  • Incidents carry version, site, batch and time metadata.
  • Rates per robot, not raw counts.
  • The suspected cause is tested by intervention (rollback or fix) when possible.

6. Architecture

incidents + fleet inventory ─▶ rates per group ─▶ factor that separates the rates across other factors ─▶ hypothesis ─▶ intervention

7. Implementation

  1. Collect incidents with metadata and the fleet inventory.
  2. Compute rates per robot by each factor (below).
  3. Check each factor across the others.
  4. Form a hypothesis and test it by rolling back or fixing a subset.
  5. Confirm the rate falls.
from collections import Counter
incidents = [("1.4.2", "siteA"), ("1.5.0", "siteA"), ("1.5.0", "siteB"), ("1.5.0", "siteB"), ("1.5.0", "siteA"), ("1.4.2", "siteB"), ("1.5.0", "siteC"), ("1.5.0", "siteB")]
fleet = {("1.4.2", "siteA"): 10, ("1.5.0", "siteA"): 6, ("1.5.0", "siteB"): 7, ("1.4.2", "siteB"): 12, ("1.5.0", "siteC"): 8}
c = Counter(incidents)
print(f"{'version':8s}{'site':7s}{'robots':>7s}{'incidents':>10s}{'per robot':>10s}")
for key, n in fleet.items():
    print(f"{key[0]:8s}{key[1]:7s}{n:7d}{c[key]:10d}{c[key] / n:10.2f}")
by_version = {v: sum(c[k] for k in fleet if k[0] == v) / sum(n for k, n in fleet.items() if k[0] == v) for v in ("1.4.2", "1.5.0")}
by_site = {s: sum(c[k] for k in fleet if k[1] == s) / sum(n for k, n in fleet.items() if k[1] == s) for s in ("siteA", "siteB", "siteC")}
print("incidents per robot by version:", {k: round(v, 2) for k, v in by_version.items()})
print("incidents per robot by site:", {k: round(v, 2) for k, v in by_site.items()})
print("version 1.5.0 is the common factor across sites; site B alone is not (1.4.2 robots at site B are fine)")

Printed (checked by the site’s tests):

version site    robots incidents per robot
1.4.2   siteA       10         1      0.10
1.5.0   siteA        6         2      0.33
1.5.0   siteB        7         3      0.43
1.4.2   siteB       12         1      0.08
1.5.0   siteC        8         1      0.12
incidents per robot by version: {'1.4.2': 0.09, '1.5.0': 0.29}
incidents per robot by site: {'siteA': 0.19, 'siteB': 0.21, 'siteC': 0.12}
version 1.5.0 is the common factor across sites; site B alone is not (1.4.2 robots at site B are fine)

Reading it: version 1.5.0 has 0.29 incidents per robot against 0.09 for 1.4.2, and it is worse in every site where both versions run, whereas site B on its own is only mildly higher (0.21 against 0.19 and 0.12): the common factor is the version, not the site. The counts are illustrative.

8. Real examples

  • After a rollout: incidents per robot doubled on the new version at all sites; the rollout was paused and rolled back on a subset to confirm.

9. Failure scenarios

  • Blaming the site with the most incidents because it has the most robots.
  • Not recording the version in incidents.
  • Changing several things during the investigation.

10. Troubleshooting

  1. Normalise by group size.
  2. Cross the factors.
  3. Check for time effects such as a network outage.

11. Validation

  • The analysis prints the lines shown (executed by the site tests).
  • After the intervention the incident rate falls in the affected group.

12. Engineering checklist

13. Further references

  • Elementary rate analysis; for formal testing use appropriate statistical tests for count data.

Related on FixTheRobot