UAV Systems Engineering · M4-L6 · Hands-on lab (runs on your machine)

Evaluate mapping accuracy

A map is only as good as its measured distance to the truth: quantify accuracy (how far map points are from the reference) and completeness (how much of the reference the map covers).

SLAM and Autonomous Localization
Established engineering
About 60 minutes

What you will do

You will compute accuracy and completeness of a LiDAR map against a reference scan or model, first on a synthetic map with a known error, then on the map from your FAST-LIO2 or LIO run (lab “Build a 3D point-cloud map”).

Before you start

A map point cloud and a reference (surveyed scan, CAD model sampled to points, or a higher-quality map), aligned to the same frame.
Python with NumPy; CloudCompare or Open3D for visual inspection.

Reference implementation and self-test

import numpy as np
rng = np.random.default_rng(0)

def nn_dist(A, B, chunk=2000):                                        # distance from each point of A to its nearest neighbour in B (brute force, chunked)
    out = np.empty(len(A))
    for i in range(0, len(A), chunk):
        d2 = ((A[i:i + chunk, None, :] - B[None, :, :]) ** 2).sum(-1); out[i:i + chunk] = np.sqrt(d2.min(1))
    return out

def accuracy_completeness(map_pts, ref_pts, tau=0.10):
    acc = nn_dist(map_pts, ref_pts)                                    # map -> reference: accuracy (how far map points are from the truth)
    comp = nn_dist(ref_pts, map_pts)                                   # reference -> map: completeness (is every reference region covered?)
    return {"accuracy_rms": float(np.sqrt(np.mean(acc ** 2))), "accuracy_median": float(np.median(acc)), "accuracy_p95": float(np.percentile(acc, 95)),
            "completeness_pct": float(100 * np.mean(comp < tau)), "precision_pct": float(100 * np.mean(acc < tau))}

def box_surface(n, size=(8, 6, 3)):
    P = rng.uniform(0, 1, (n, 3)) * size; face = rng.integers(0, 3, n)
    for k in range(3): m = face == k; P[m, k] = np.where(rng.random(m.sum()) < 0.5, 0.0, size[k])
    return P
ref_pts = box_surface(4000)
good = ref_pts + 0.02 * rng.standard_normal(ref_pts.shape)                   # a good map: 2 cm noise
good = good[rng.random(len(good)) < 0.7]                                      # and only 70 % of the points were observed
bad = box_surface(2500) + 0.06 * rng.standard_normal((2500, 3)) + np.array([0.0, 0.0, 0.25])   # a distorted map: 6 cm noise and a 25 cm vertical offset
r_good, r_bad = accuracy_completeness(good, ref_pts), accuracy_completeness(bad, ref_pts)
print("good map:", {k: round(v, 3) for k, v in r_good.items()}); print("bad map: ", {k: round(v, 3) for k, v in r_bad.items()})
assert r_good["accuracy_rms"] < 0.05 and r_good["completeness_pct"] > 75 and r_bad["accuracy_rms"] > 0.15 and r_bad["precision_pct"] < r_good["precision_pct"] - 30
assert accuracy_completeness(ref_pts, ref_pts)["accuracy_rms"] < 1e-12

Accuracy and completeness of a map against a reference, tested on good and distorted synthetic maps.

Steps

Align the map to the reference (coarse by known transform, then ICP with a tight threshold). Report the alignment residual: it is part of the error budget.
Downsample both clouds with the same voxel size so density differences do not bias the statistics.
Compute accuracy (map to reference), completeness (reference to map) and outlier fraction with the code above, with $\tau$ chosen from your requirement (for example 10 cm).
Produce a colour-coded error map in CloudCompare (cloud-to-cloud distance) and look for systematic patterns: scale error, bending, double walls.
Relate the findings to the trajectory metrics of the previous lab: bending of long walls points to heading drift.

Questions to answer

Why must you report both accuracy and completeness?
What does a double wall in the error map tell you about the odometry?
What accuracy does your mission require of the map (lesson M4-T14)?

How to know it worked

The self-test passes; you report accuracy, completeness and the threshold $\tau$ .
You included the alignment residual in the report.
You identified one systematic error pattern in your own map, or stated that none was visible.

If it does not work

Troubleshooting
Symptom	Likely cause	What to check
Accuracy dominated by misalignment	Coarse alignment poor	Refine with ICP using inliers only; report the residual
Completeness low	Reference covers regions the vehicle never saw	Crop the reference to the observed region
Brute force too slow	Large clouds	Use a k-d tree (scipy, Open3D) or voxel downsample

References

Record your progress

Saved in this browser. This lab runs on your own machine, so the site cannot check it for you.