Skip to the lesson
FixTheRobot Explore Robotics / Navigation / M1 Navigation Foundations / M1-T6

Navigation · Theory · M1

M1-T6 — Measuring Navigation: Success, Efficiency, Safety and Comfort

“It worked on my desk” is not a result. Four families of numbers tell you whether a robot navigated well, and which part to fix when it did not.

  • Lesson 6 of 6
  • Module 1: Navigation Foundations
  • Theory
  • About 9 minutes to read

Two robots can both reach the goal in every test. One takes the shortest route, keeps a hand’s breadth from every wall and glides. The other wanders, scrapes the door frame and lurches. A single yes-or-no hides all of that. This lesson builds a small set of metrics that separate them, shows how many runs you need before a number means anything, and gives a way to name the stage that failed.

Learning objectives

After this lesson, you should be able to:

  • Compute success rate, SPL, minimum clearance, near-miss count and RMS jerk from a logged run.
  • Explain what each metric hides, and choose a set that covers effectiveness, efficiency, safety and comfort.
  • Judge how many runs are needed to trust a success rate, using a confidence interval.
  • Classify a failed run by the navigation stage that caused it, from its log and trace.

The intuition

Think of how a driving examiner marks a test. Did you reach the destination? (success) Did you take a sensible route? (efficiency) Did you stay clear of kerbs and other cars? (safety) Was the ride smooth for a passenger? (comfort) A pass needs all four, and a fail comes with a reason.

FamilyQuestionCommon metrics
EffectivenessDid it get there?Success rate, mission completion, time to goal
EfficiencyDid it get there sensibly?Path length, SPL, mean speed, replans per run
SafetyDid it get there without risk?Collisions, minimum clearance, near misses, emergency stops
Comfort and mannersWould you ride, or work, beside it?RMS jerk, speed changes, personal-space intrusions

No single number covers all four. A robot can score 100% success and still be unusable.

The mathematics

Success and path efficiency

Let a run i have success Si (1 if it reached the goal within tolerance and time, else 0), shortest-path length li from the start to the goal on the map, and driven path length pi. The success rate is the mean of Si. Success weighted by Path Length (SPL) rewards reaching the goal by a route close to the shortest:

Success weighted by Path Length
SPL=(1N) ∑i Si×limax⁡(pi, li)\mathrm{SPL}=\frac{(\frac{1}{N})\,\sum _{i}\,S_{i}\times l_{i}}{\max (p_{i},\,l_{i})}

It is 1 for a perfect run, falls as the path gets longer than necessary, and is 0 for a failure. The max stops a measurement error that makes p smaller than l from giving a score above 1.

Clearance

For a round robot of radius rrobot and an obstacle disc of radius robs centred at c, the clearance at a position p is the gap between their edges:

Clearance
clearance⁡(p)=min⁡ over obstacles[distance⁡(p, c)−robs−rrobot]\operatorname{clearance}(p)=\min \,\text{over obstacles}[\operatorname{distance}(p,\,c)-r_{\text{obs}}-r_{\text{robot}}]
min⁡clearance=min⁡ over the run of clearance⁡(p)\min _{\text{clearance}}=\min \,\text{over the run of}\,\operatorname{clearance}(p)
collision if clearance⁡(p)<0\text{collision if}\,\operatorname{clearance}(p)<0
near miss if clearance⁡(p)<dnear\text{near miss if}\,\operatorname{clearance}(p)<d_{\text{near}}

A near-miss threshold dnear of 0.15 to 0.30 m is typical for indoor robots, and should come from the requirements. A run with min_clearance = 0.04 m succeeded and was a near collision.

Comfort: jerk

Acceleration a is the rate of change of speed and jerk j is the rate of change of acceleration. Sampled every Δt,

Jerk from a speed profile
ak=vk−vk−1Δta_{k}=\frac{v_{k}-v_{k-1}}{\Delta t}
jk=ak−ak−1Δtj_{k}=\frac{a_{k}-a_{k-1}}{\Delta t}
RMS jerk=mean of jk2\mathrm{RMS}\,\text{jerk}=\sqrt{\text{mean of}\,j_{k}^{2}}

High jerk shakes cargo and passengers and wears hardware. People notice it before they notice anything else.

How many runs?

A success rate measured on n runs is itself uncertain. For k successes in n runs, the Wilson score interval at 95% confidence is

Wilson interval
p^=kn\hat{p}=\frac{k}{n}
centre=p^+z22n1+z2n\text{centre}=\frac{\hat{p}+\frac{z^{2}}{2n}}{1+\frac{z^{2}}{n}}
half−width=z p^(1−p^)n+z24n21+z2n\text{half}-\text{width}=\frac{z\,\sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z^{2}}{4n^{2}}}}{1+\frac{z^{2}}{n}}
z=1.96z=1.96
Confidence interval narrows with more runsFive vertical bars, one for each of 10, 20, 50, 100 and 200 runs, all with a measured success rate of 90 percent. The bars are widest for 10 runs, from 60 to 98 percent, and narrow to 85 to 93 percent for 200 runs.00.250.50.751 102050100200 number of runs (each with a measured success rate of 90 %)plausible true success rate
A 90% success rate measured on more runs gives a narrower range of plausible true values. Ten runs cannot tell a good robot from a mediocre one.
Si
success of run i, 0 or 1.
li, pi
shortest and driven path lengths, in metres.
dnear
the distance below which a pass counts as a near miss.
Δt
the time between samples of speed.

A worked example

SPL on four runs. Scout makes four trips. The shortest path l, the driven path p, and success S are:

RunSl (m)p (m)S × l / max(p, l)
111012.510 / 12.5 = 0.800
21121212 / 12 = 1.000
308200 (it failed)
41203020 / 30 = 0.667
Success rate and SPL
success rate=34=75 %\text{success rate}=\frac{3}{4}=75\,\%
SPL=0.800+1.000+0+0.6674=2.4674=0.617\mathrm{SPL}=\frac{0.800+1.000+0+0.667}{4}=\frac{2.467}{4}=0.617

The success rate of 75% hides that two of the three successes took routes 25% to 50% longer than necessary. SPL of 0.617 says so.

A near miss that success hides. On run 4 the robot passes a pillar of radius 0.30 m with its centre 0.64 m from the pillar’s centre. Scout’s radius is 0.30 m. The clearance is 0.64 − 0.30 − 0.30 = 0.04 m. The run counts as a success, its SPL is fine, and nothing in either number says that Scout nearly hit the pillar. Only the clearance does.

Jerk of a trapezoidal stop. Scout’s speed, sampled once a second, is 0, 0.5, 1.0, 1.0, 0.5, 0 m/s. Accelerations are +0.5, +0.5, 0, −0.5, −0.5 m/s², and jerks are 0, −0.5, −0.5, 0 m/s³. RMS jerk = √( (0 + 0.25 + 0.25 + 0) / 4 ) = √0.125 = 0.354 m/s³. Smoothing the speed profile with a ramp of the same total duration but gentler corners would lower it. Module 5 builds such profiles.

How many runs? Scout succeeds in 18 of 20 trials, 90%. The 95% Wilson interval is [0.699, 0.972]: the true rate could plausibly be anywhere from 70% to 97%. With 180 of 200 the same 90% gives [0.851, 0.934]. A claim of “over 95% reliable” needs far more runs than 20 to support it, and an improvement from 88% to 90% cannot be seen at all in 20 runs.

Engineering implications and limitations

Fix the scenarios, then compare

Run every version of the software on the same set of scenarios with fixed random seeds, so a change in the number is a change in the software and not in the luck of the draw. Record the scenario, the software version and the full log with every run.

Report distributions, not just means

The average time to goal hides the one run that took ten minutes. Report the median, the 95th percentile and the worst case, and list every failure with its cause.

Name the failing stage

Classify each failure by the stage that caused it so that effort goes where it pays. The signatures below are typical:

StageWhat the log and trace show
SensingObstacles appear and vanish between scans; missing returns on glass or black surfaces; timestamps out of order
LocalizationThe pose jumps, covariance grows, scans no longer match the map, the path appears to move relative to the walls
Mapping / costmapObstacles remain after they have gone; walls missing from the costmap; the planner and controller disagree about a gap
PlanningNo path found, or a path through a gap the footprint cannot clear, or a path much longer than needed
ControlWeaving, corner cutting, overshoot, saturated commands, a robot that cannot hold the path at speed
RecoveryThe same recovery repeated with the same result, or no recovery attempted

Beware of gaming a metric

A robot that crawls never collides. A robot that always takes a shortcut through people has a perfect SPL. Use metrics in sets, and put hard limits (zero collisions, a minimum clearance) beside the averages.

Common misconceptions

“A high success rate means the robot navigates well.”

It says the robot reached the goal. It says nothing about how close it came to hitting something, how long it took, or how much it shook.

“SPL measures safety.”

It measures route efficiency among the successful runs. A run that clips a wall and reaches the goal by the shortest route scores a perfect SPL.

“Ten good runs prove it works.”

Ten runs with nine successes is compatible with a true success rate between 60% and 98%. More runs, or a confidence interval, are needed to say anything.

“If the robot did not reach the goal, the planner failed.”

Any stage can cause the failure. Classify it from the evidence before changing the planner.

Engineering takeaways

  1. Measure effectiveness, efficiency, safety and comfort. No single number covers them.
  2. SPL = mean of S × l / max(p, l), and success weighting makes failures count as zero.
  3. Clearance is measured edge to edge, and a successful run can be a near miss.
  4. Jerk is the rate of change of acceleration; RMS jerk summarizes the roughness of a speed profile.
  5. Use enough runs to make the interval narrow, run the same scenarios each time, and classify every failure by stage.

References and further reading

The paper that defined SPL and general guidance on evaluation.

  • P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva and A. R. Zamir, “On Evaluation of Embodied Navigation Agents”, arXiv:1807.06757, 2018.
  • E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference”, Journal of the American Statistical Association, 22(158), 1927.
  • A. Francis et al., “Principles and Guidelines for Evaluating Social Robot Navigation Algorithms”, ACM Transactions on Human-Robot Interaction, 2025.
  • Nav2 documentation, “Navigation Concepts” and the metrics tooling, docs.nav2.org.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A robot has a success rate of 100% over 10 runs but a minimum clearance of 0.03 m in two of them. What does this tell you and what should you do?

Explanation

The runs reached the goal but passed obstacles with almost no margin. Treat them as near collisions and investigate before deploying: inflation, footprint or controller tuning may be too tight. Success rate and SPL cannot show this; clearance does.

Question 2

Three of four runs succeed, with driven lengths 12, 15 and 9 m against shortest paths of 10, 15 and 9 m, and the fourth fails. What is the SPL?

Explanation

The terms are 10/12 = 0.833, 15/15 = 1.000, 9/9 = 1.000 and 0 for the failure, so SPL = (0.833 + 1 + 1 + 0)/4 = 0.708. Success rate is 75%.

Question 3

Why can 18 successes in 20 runs not show that a change improved the success rate from 88% to 90%?

Explanation

With n = 20 the 95% interval around 90% is about 70% to 97%, far wider than the 2 point difference. Distinguishing 88% from 90% needs thousands of runs, or a much larger improvement.

What’s next

That completes the foundations. Next: maps and costmaps, the representation every planner searches.

Practice this lesson

Score a Navigation Run

Compute success, SPL, clearance and jerk for a logged run, and catch the metric that hides a near miss.

Next lesson

M2-T1 — Occupancy Grids for Planning

Choose a grid resolution and a policy for unknown space, and explain what each choice costs the planner.

Nothing below is required to finish this lesson.

  • Before
    M1-T5: Configuration Space

    Turn a robot with a footprint among obstacles into a point among inflated obstacles, and say when that simplification fails.

  • Lab
    M1-L3: Score a Navigation Run

    Compute success, SPL, clearance and jerk for a logged run, and catch the metric that hides a near miss.