Validating navigation in a real environment

1. Engineering problem

Navigation that works in three demonstrations is not validated. Success rates need many trials, and the result depends on the route, the people around, the time of day and the robot’s battery.

2. When is this needed?

• Before releasing a robot to a site.
• After any change to the stack or the environment.
• When comparing two configurations.

3. Selection and design decisions

Run each route many times in representative conditions and record goal reached or not, time, path length, interventions, recoveries and goal position error. For a success rate s out of n trials, the Wilson score interval gives a 95 % interval that behaves well for small n and rates near 100 %.

4. Advantages and disadvantages

• More trials tighten the interval and cost time; a few hundred runs may be needed for 99 % claims.
• Staged scenarios are repeatable; live traffic is realistic.
• Metrics must be defined before the trials.

5. Requirements and constraints

• The goal for success is defined in advance (tolerance, time limit).
• Trials cover the ODD’s conditions.
• Failures are analysed, not discarded.

6. Architecture

routes × conditions × repeats ─▶ log outcomes ─▶ success rate with interval, goal error, recoveries ─▶ fix and repeat

7. Implementation

1. Define success and failure.
2. Choose routes and conditions from the ODD.
3. Run repeated trials and log them.
4. Compute the success rate and its interval (below).
5. Analyse each failure and fix, then repeat the full set.

import math
def wilson(s, n, z=1.96):
    p = s / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h
for s, n in ((18, 20), (95, 100), (20, 20)):
    lo, hi = wilson(s, n)
    print(f"{s}/{n} goals reached: {s / n * 100:.0f} %, 95 % interval {lo * 100:.0f} to {hi * 100:.0f} %")
errs = [0.04, 0.07, 0.05, 0.12, 0.06, 0.09, 0.05, 0.08]
errs.sort()
print("goal position error: median %.3f m, worst %.2f m" % ((errs[3] + errs[4]) / 2, errs[-1]))

Printed (checked by the site’s tests):

18/20 goals reached: 90 %, 95 % interval 70 to 97 %
95/100 goals reached: 95 %, 95 % interval 89 to 98 %
20/20 goals reached: 100 %, 95 % interval 84 to 100 %
goal position error: median 0.065 m, worst 0.12 m

Reading it: 18 of 20 reaches is 90 % with a 95 % interval of about 70 to 97 %; 95 of 100 is 95 % (89 to 98 %); even 20 of 20 only supports 84 % or better. The goal error list gives a median of 0.065 m and a worst case of 0.12 m. The trial results are illustrative.

8. Real examples

• Hospital delivery route: 200 runs across shifts, with the interval reported per route.

9. Failure scenarios

• Reporting a success rate from five runs.
• Dropping failed runs as “operator error”.
• Changing parameters between trials without recording them.

10. Troubleshooting

1. Group failures by route and condition.
2. Check the recovery counts: many recoveries with success still hide a weak configuration.
3. Compare before and after changes with the same trial set.

11. Validation

• The example prints the lines shown (executed by the site tests).
• Success rate and interval meet the requirement on the full condition set.

12. Engineering checklist

• Success defined
• Routes and conditions from the ODD
• Enough trials
• Interval reported
• Failures analysed
• Versions recorded

13. Further references

• The Wilson score interval is a standard binomial confidence interval; see statistics texts for the derivation.

Related on FixTheRobot