M8-T8 — SLAM Evaluation and Benchmarking

Learning objectives

After this lesson, you should be able to:

• Define ATE and RPE, say what each measures, and compute them for a small example.
• Explain the effect of the alignment (none, SE(3), Sim(3)) and of time association on reported error.
• Summarise repeated runs with medians, percentiles and failure rates, not only means.
• Read a benchmark table critically, identifying the protocol choices that make numbers comparable or not.

The problem and the intuition

Two students report the accuracy of the same SLAM run. One says 15 cm, the other 6 cm, a third says 0.5 cm. All three are right, because the first reports raw error against ground truth in a different frame, the second first aligned the trajectories with a rigid transform, and the third also fitted a scale. The question “how accurate is this SLAM?” has no answer until you say what is being compared with what, after which transformation, over how many runs, and what counts as a failure.

The aim of evaluation is to answer a decision question: will this system meet my requirement on my robot? That needs the right metric for the use. Odometry used for local control needs low relative error over short distances; a map used for navigation needs low absolute error over the whole map; a system used in a fleet needs a low failure rate, which no average error reveals.

Always report: the metric, the alignment, the time association, the number of runs and the failures. A single ATE number without those is not a result.

Metrics, alignment and statistics

Problem. Quantify how close an estimated trajectory and map are to ground truth, in a way that is fair, reproducible and relevant to the application.

Assumptions. Ground truth of adequate accuracy exists (motion capture, RTK-GNSS, total station, laser tracker, or a survey map); timestamps of estimate and truth can be associated; the frames of the two may differ by an unknown transformation.

Absolute trajectory error (ATE)

After aligning the estimated positions p̂_i to the ground-truth positions p_i by the best transformation S (rigid in SE(3), or a similarity in Sim(3) for monocular systems), the error is the RMSE of the remaining position differences (Sturm et al., 2012):

ATE measures global consistency: a single early rotation error that bends the whole trajectory raises it everywhere.

Relative pose error (RPE)

For pairs of poses separated by Δ (a number of frames, a time or a distance), compare the estimated relative motion with the true one:

where Q are ground-truth and P estimated poses. RPE needs no alignment, measures local drift, and depends strongly on Δ (Sturm et al., 2012). The KITTI odometry benchmark (Geiger et al., 2012) reports translational error as a percentage of distance and rotational error in degrees per metre averaged over sub-sequences of 100 m to 800 m. Zhang and Scaramuzza (2018) give recommendations for visual(-inertial) odometry, including the choice of alignment and sub-trajectory length.

What the alignment hides

Time association and ground truth quality

Estimates and truth are sampled at different times. Pair each estimate with the nearest ground-truth sample within a maximum difference (0.01 to 0.02 s is common) or interpolate. Without it, motion during a timestamp offset appears as error (M6-T8). Ground truth should be several times more accurate than the errors being measured.

Statistics over runs

• Repeat. SLAM is stochastic (RANSAC, threading, random seeds). Run each configuration several times and report the median and spread (inter-quartile range or percentiles).
• Count failures. A run that loses tracking or diverges has no meaningful error. Report the failure rate separately; do not average a catastrophic 30 m error into a mean of 20 cm, and do not drop it.
• Report drift as a rate (percent of distance, degrees per 100 m) for odometry, so that results over different path lengths compare.
• Report cost: CPU, GPU, memory, latency (M7-T8).

Other metrics

• Map accuracy and completeness: distance from reconstructed points to a ground-truth surface, and the fraction of the true surface that is covered (for dense mapping).
• Consistency: the normalised estimation error squared (NEES) compares error with the claimed covariance; its average should equal the state dimension (M2-T5, M6-T7).
• Loop-closure precision and recall (M5-T10): the fraction of accepted loops that are correct, and of true loops found.
• Task metrics: navigation success rate and path efficiency on the map, which is what the application needs.

ATE

absolute trajectory error after alignment (global consistency).

RPE(Δ)

relative pose error over interval Δ (local drift).

S

alignment transform estimated from the trajectories themselves.

NEES

normalised estimation error squared: eᵀP⁻¹e.

A worked example: the same run, three numbers

1. The data. Ground truth: ten poses along a straight line, p_i = (i, 0) for i = 0 … 9 (metres). Estimate: p̂_i = (1.02 i, 0.02 i): a drift of 2% in scale and a small heading error of about 1.1°.
2. No alignment. The differences are (0.02 i, 0.02 i), whose norms are 0.0283 i. The RMSE over i = 0 … 9 is 0.0283 × √(Σi²/10) = 0.0283 × √28.5 = 0.151 m.
3. Rigid alignment (translation + rotation). The best rigid fit removes the heading offset and centres the trajectories. The remainder is the scale error: ATE = 0.058 m.
4. Similarity alignment (adding scale). The estimate is exactly a scaled, rotated copy, so ATE ≈ 0 and the fitted scale is 0.98. The alignment has absorbed the entire error; for a metric sensor that would be a mistake, since the scale error is real. Report the fitted scale (here 0.98, i.e. 2%).
5. RPE. Over Δ = 1 step, the relative translation error is the difference of increments: (0.02, 0.02), norm 0.028 m. Over Δ = 5 steps it is 0.141 m. RPE grows with Δ, and an RPE quoted without Δ is incomplete. As a percentage of distance: 0.141 / 5 = 2.8%, comparable between intervals.
6. Repeated runs. Ten runs give ATE (m): 0.21, 0.19, 0.23, 0.20, 0.22, 0.18, 0.24, 0.20, 2.80, 0.21. The mean is 0.47 m, the median 0.21 m, the 90th percentile 0.50 m, and one run in ten failed (ATE above 1 m). “0.47 m” describes no run: the honest statement is “median 0.21 m, 10% failures, one catastrophic at 2.8 m”.

Engineering implications and limitations

Engineering implications

• Choose the metric from the requirement. A local controller needs RPE at the control horizon; a map for navigation needs ATE and map accuracy; a fleet needs failure rates.
• State the protocol in every table: alignment type, Δ for RPE, maximum time difference, number of runs, how failures are counted, hardware.
• Use established tools (evo, rpg_trajectory_evaluation, the KITTI development kit) so numbers are comparable, and check them against a hand calculation on a small example, as in this lesson. Record the command with its options next to the result.
• Keep raw trajectories and logs. Summary numbers cannot be re-analysed; trajectories can.
• Test under controlled failures (M8-L4): noise, outliers and dropouts, to see how errors degrade instead of reporting only the clean case.
• Validate ground truth. A motion-capture system can have its own errors and latency; a GNSS ground truth is poor indoors.

Formats, tools and the alignment to choose

Every benchmark has its own trajectory file, and most wrong numbers in comparisons come from reading one as another. The three used in this lesson:

The tool evo reads all three, and ROS bags, and provides evo_ape (absolute pose error), evo_rpe (relative pose error), evo_traj (inspect and plot trajectories) and evo_res (compare saved results). A typical call names the format, the reference and the estimate, then the options that fix the protocol: alignment, scale correction, the time association, and for RPE the interval and its unit. Whatever the tool, the options chosen belong in the results table.

The alignment is chosen from the sensor, not from what gives the smallest number (Zhang and Scaramuzza, 2018):

Two further traps. Aligning with only the first pose instead of the whole trajectory measures drift from the start; aligning on all poses spreads it, and the two give different numbers. And a ground-truth file with different time stamps from the estimate must be associated by time before any metric is computed: nearest-neighbour association with no limit on the gap pairs poses that are tens of milliseconds apart, which at 1 m/s is centimetres of error that belongs to neither system.

Limits

No single metric captures all that matters. ATE is dominated by a few early errors; RPE ignores global consistency; both depend on the alignment and the path. Public benchmarks are valuable but finite: systems can be tuned to them, and results on indoor sequences say little about outdoor or long-term operation. Evaluation of maps (as opposed to trajectories), of uncertainty, and of long-term and multi-robot systems has fewer accepted protocols. Aggregate statistics from few runs have wide uncertainty that is rarely reported.

Common misconceptions

Engineering takeaways

1. ATE (after alignment) measures global consistency; RPE(Δ) measures local drift; both need their protocol stated.
2. Alignment choice changes the number: none 0.151 m, rigid 0.058 m, Sim(3) ≈ 0 for the same estimate; report the fitted scale.
3. Report median, spread and failure rate over repeated runs; do not average failures into a mean or drop them.
4. Time association and ground-truth quality limit what can be measured.
5. Read benchmark tables for protocol first, numbers second.

References and further reading

Primary papers and standard texts for the metrics and benchmarks discussed.

• Sturm, J., Engelhard, N., Endres, F., Burgard, W. and Cremers, D. (2012). A benchmark for the evaluation of RGB-D SLAM systems. IEEE/RSJ IROS.
• Geiger, A., Lenz, P. and Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. IEEE CVPR.
• Grupp, M. (2017). evo: Python package for the evaluation of odometry and SLAM. github.com/MichaelGrupp/evo: the formats and tools named in this lesson.
• Burri, M. et al. (2016). The EuRoC micro aerial vehicle datasets. International Journal of Robotics Research 35(10).
• Zhang, Z. and Scaramuzza, D. (2018). A tutorial on quantitative trajectory evaluation for visual(-inertial) odometry. IEEE/RSJ IROS.
• Horn, B. K. P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A 4(4).
• Umeyama, S. (1991). Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 13(4).
• Bar-Shalom, Y., Li, X. R. and Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation. Wiley (NEES and consistency).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

To close the theory of the track, the last lesson of this module surveys what is established and what is open, and how to read the current research with care.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T7: Neural and Implicit Mapping

  Explain NeRF- and Gaussian-splatting-based SLAM, how they differ from classical maps and their limits.

• Lab
  M8-L2: Compare SLAM Trajectories Using Evaluation Metrics

  Implement trajectory alignment, ATE and RPE and rank three SLAM runs fairly.