FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 19

Engineering Guides · Module 19

Production deployment and rollback

Roll out to a few robots first, check health, widen only if healthy, and keep the previous version ready.

  • Intermediate
  • Module 19: Testing, Validation & Field Deployment
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

A software update that works on one robot can fail on others because of site differences, maps or hardware variants. A staged rollout limits the damage; a tested rollback ends it quickly.

2. When is this needed?

  • Releasing a new software version or configuration.
  • Updating maps.
  • Designing the update system.

3. Selection and design decisions

Staged rollout (canary): update a small fraction of the fleet, check health metrics against a threshold, widen to a larger fraction, and so on; on a failed check, roll back to the previous version. Keep software, configuration and map versions compatible: record which software versions need which map and configuration schema versions, and deploy dependencies first. Artifacts are verified with a checksum before installation (see the fleet OTA guide).

StageFractionCheck
CanaryAbout 2 % (at least one robot)Boot, localisation, a standard route, no new faults
EarlyAbout 10 %Fault rates and task success against the baseline
HalfAbout 50 %Same, plus performance
All100 %Continue monitoring
RollbackAny stagePrevious version restored automatically on failure

4. Advantages and disadvantages

  • Slow rollouts are safer and delay fixes; fast ones are the reverse.
  • Automatic rollback needs reliable health signals; manual rollback is slower and surer to be deliberate.
  • Keeping two versions on each robot costs storage.

5. Requirements and constraints

  • Health checks defined before the rollout.
  • Previous version, configuration and map kept on every robot.
  • Rollback tested at least once before it is needed.

6. Architecture

build ─▶ staging tests ─▶ canary ─▶ widen (health checks) ─▶ all robots
                              failed check ─▶ rollback to previous version

7. Implementation

  1. Define the health checks and thresholds.
  2. Verify compatibility of software, configuration and map.
  3. Roll out in stages (below), checking between stages.
  4. Roll back on failure and analyse.
  5. Record the version on each robot.
def rollout(stages, health):
    log = []
    for frac, ok in zip(stages, health):
        if not ok:
            log.append(f"{frac * 100:.0f} % of the fleet: health check FAILED -> roll back to the previous version")
            return log
        log.append(f"{frac * 100:.0f} % of the fleet: health check passed")
    log.append("rollout complete")
    return log
print("\n".join(rollout([0.02, 0.1, 0.5, 1.0], [True, True, True, True])))
print()
print("\n".join(rollout([0.02, 0.1, 0.5, 1.0], [True, False, True, True])))
versions = {"1.4.2": "map v7", "1.5.0": "map v8"}
print("\nsoftware and map compatibility:", versions, "-> deploy the map with, or before, the software that needs it")

Printed (checked by the site’s tests):

2 % of the fleet: health check passed
10 % of the fleet: health check passed
50 % of the fleet: health check passed
100 % of the fleet: health check passed
rollout complete

2 % of the fleet: health check passed
10 % of the fleet: health check FAILED -> roll back to the previous version

software and map compatibility: {'1.4.2': 'map v7', '1.5.0': 'map v8'} -> deploy the map with, or before, the software that needs it

Reading it: in the first run every stage passes and the rollout completes; in the second the check at the 10 % stage fails and the rollout stops with a rollback. The compatibility line reminds that software 1.5.0 needs map v8: deploy the map with or before the software. The versions and fractions are illustrative.

8. Real examples

  • Warehouse fleet: two robots first, then a zone, then all; a drop in task success at the zone stage triggers the rollback.

9. Failure scenarios

  • Updating all robots at once.
  • Software that needs a new map pushed before the map.
  • Rollback never tested.

10. Troubleshooting

  1. Compare health metrics between old and new versions at each stage.
  2. Check version compatibility on the failing robots.
  3. Check that the rollback really restores the previous version.

11. Validation

  • The simulation prints the lines shown (executed by the site tests).
  • A rollback drill restores the previous version on a test robot within the target time.

12. Engineering checklist

13. Further references

  • Staged rollout is common deployment practice; see the OTA guide for artifact verification.

Related on FixTheRobot