M5-T4 — Identifying Noise Parameters from Data

Learning objectives

After this lesson, you should be able to:

• Design experiments that isolate each odometry noise parameter, and estimate them by moment matching.
• State the accuracy of a variance or standard-deviation estimate from n trials, and choose n for a required accuracy.
• Fit a mixture of good readings and outliers with the EM algorithm, and derive its two steps from Bayes’ rule.
• Validate a fitted noise model on held-out data with coverage and whiteness checks.

The intuition

A noise parameter is the answer to a question about the robot: how far off does it end up after a 10 m drive? How much does a 90° turn vary? What fraction of lidar readings are wrong? Each question has an experiment, and the experiment needs a reference, something that knows the truth better than the robot does: a tape measure, a motion-capture system, a surveyed wall, a total station, or the stop of the robot against a fixed object.

The recipe is the same for all of them:

1. Isolate one effect. A turn on the spot isolates $\alpha_1$. Straight drives of different lengths separate the along-track error ($\alpha_3$) from the sideways error ($\alpha_2$) and show how each grows with distance. A sensor looking at a fixed wall isolates its noise.
2. Repeat the experiment many times and record the error against the reference.
3. Estimate the parameter from the errors with the formula that connects it to a measurable variance.
4. State the error bar. An estimate of a variance from $n$ trials is itself uncertain, by about $\sqrt{2/n}$ of its value.
5. Validate on data that were not used for the fit.

The left panel is a sobering one. To know a variance within 10% takes about 200 trials, and within 5% about 800. A calibration made from five runs of each distance (a typical afternoon) pins a noise parameter to roughly ±30%. That is usually acceptable, since a filter does not need the parameters to better than a factor of 1.5, but it must be known: quoting $\alpha_3 = 0.00043$ to two digits from 60 trials claims a precision that the data do not have.

Many sensors also produce outliers, and in real logs nobody has labelled which readings are wrong. The two panels on the right show a way through: an expectation-maximization (EM) fit of the mixture of the sensor lesson. It starts from a poor guess (a noise of 0.3 m and half of the readings being outliers) and, in five iterations, recovers the right centre, spread and outlier share, without a single label.

A parameter without an error bar is a guess. The relative error of a variance estimate is √(2/n): plan the number of trials from the accuracy you need.

Definitions

Estimator

A rule that turns data into an estimate of a parameter. Its own scatter over repetitions of the experiment is its standard error.

Moment matching

Choosing the parameter so that a model moment (a variance, say) equals the corresponding moment of the data.

Reference (ground truth)

An independent measurement much more accurate than the thing being calibrated.

Responsibility $r_i$

In a mixture, the probability that reading $i$ came from the good (Gaussian) component, given all the current parameters.

EM (expectation-maximization)

An iteration that alternates between computing responsibilities (the E step) and re-fitting each component to the readings it is responsible for (the M step).

Coverage

The fraction of true errors that fall inside the interval a model claims (for example ±1σ should cover about 68%).

Derivation: the error bars, the odometry estimators and EM

How accurate is a variance estimate?

1. $$\dfrac{n\,\hat{\sigma}^2}{\sigma^2} = \sum_{j=1}^{n}\Bigl(\dfrac{e_j}{\sigma}\Bigr)^2 \sim \chi^2_n$$

   The sum of $n$ squared independent standard normals is chi-square distributed with $n$ degrees of freedom (the ellipse lesson, with $n$ instead of 2).

2. $$\mathrm{Var}(\chi^2_n) = 2n \;\;\Rightarrow\;\; \mathrm{Var}(\hat{\sigma}^2) = \sigma^4\,\dfrac{2n}{n^2} = \dfrac{2\sigma^4}{n}$$

   A chi-square variable has mean $n$ and variance $2n$. Scaling by $\sigma^2/n$ multiplies the variance by $\sigma^4/n^2$.

3. $$\dfrac{\mathrm{se}(\hat{\sigma}^2)}{\sigma^2} = \sqrt{\dfrac{2}{n}}$$

   The relative error of the variance. For $n = 60$ it is 18%; for 200, 10%; for 800, 5%.

4. $$\hat{\sigma} = \sqrt{\hat{\sigma}^2} \;\;\Rightarrow\;\; \mathrm{se}(\hat{\sigma}) \approx \dfrac{1}{2\sigma}\,\mathrm{se}(\hat{\sigma}^2) = \dfrac{\sigma}{\sqrt{2n}}$$

   The first-order propagation of the previous module (the derivative of the square root is $1/(2\sigma)$). The standard deviation is known to half the relative accuracy of the variance: 9% for 60 trials.

The odometry estimators

Each estimator is the experiment's mean square divided by its theoretical factor, so that its expectation is the parameter. For $d$ straight steps of 1 m, the along-track error is the sum of $d$ independent distance errors of variance $\alpha_3$ each, so its variance is $\alpha_3 d$. The sideways error is the sum, over steps, of the sideways effect of the heading errors: heading noise of the first turn of step $j$ reaches $n-j+1$ steps and that of the second turn $n-j$ steps, each of variance $\alpha_2$.

1. $$\mathrm{Var}(y_d) = \alpha_2\Bigl[\sum_{k=1}^{d} k^2 + \sum_{k=0}^{d-1} k^2\Bigr] = \alpha_2\left[\dfrac{d(d+1)(2d+1)}{6} + \dfrac{(d-1)d(2d-1)}{6}\right]$$

   The previous lesson’s sum, kept exactly: each turn’s noise $\varepsilon$ shifts the position by (number of later steps) × $\varepsilon$ (small angles, steps of 1 m).

2. $$= \alpha_2\,\dfrac{4d^3 + 2d}{6} = \alpha_2\,\dfrac{d\,(2d^2 + 1)}{3}$$

   Add the two sums. For large $d$ this is about $\tfrac{2}{3}\alpha_2 d^3$, the d^1.5 law. The factor is exactly 6 for $d = 2$, 85 for $d = 5$ and 670 for $d = 10$.

3. $$\hat{\alpha}_2 = \dfrac{1}{N}\sum_j \dfrac{y_j^2}{d_j(2d_j^2 + 1)/3}, \qquad \mathbb{E}[\hat{\alpha}_2] = \alpha_2$$

   Each term has expectation $\alpha_2$ and the same relative variance ($2$), so the plain average of the normalized terms is the best combination of trials at different distances.

EM for a good-plus-outlier mixture

If we knew which readings were outliers, the fit would be trivial: the mean and spread of the good ones, and the fraction of the others. We do not, but Bayes' rule gives each reading a probability of being good, and we can use those as soft labels.

1. $$r_i = P(\text{good} \mid z_i) = \dfrac{(1-\varepsilon)\,\mathcal{N}(z_i; \mu, \sigma^2)}{(1-\varepsilon)\,\mathcal{N}(z_i; \mu, \sigma^2) + \varepsilon\, U(z_i)}$$

   E step. Bayes’ rule (the Bayes lesson) with prior $1-\varepsilon$ for “good” and the two component densities as likelihoods.

2. $$\dfrac{\partial}{\partial \mu}\sum_i r_i \ln\mathcal{N}(z_i; \mu, \sigma^2) = \sum_i r_i\,\dfrac{z_i - \mu}{\sigma^2} = 0 \;\Rightarrow\; \mu = \dfrac{\sum_i r_i z_i}{\sum_i r_i}$$

   M step for the centre. Maximise the responsibility-weighted log-likelihood of the good component. The derivative gives the weighted mean (and, in the same way, the weighted variance for $\sigma^2$).

3. $$\varepsilon = \dfrac{1}{n}\sum_i (1 - r_i)$$

   M step for the weight. The share of readings attributed to the outlier component.

4. $$\text{each iteration does not decrease the likelihood of the data}$$

   A general property of EM (state without proof here). It converges to a local maximum, not necessarily the best one, so the starting guess matters.

Estimate a parameter by the formula that links it to a measurable variance, state its error bar (√(2/n) relative for a variance), and for unlabelled outliers let Bayes’ rule supply soft labels (EM).

A worked example

Two experiments, one program (every number below is printed by it).

1. Odometry experiment. The true robot has $\alpha_1 = 0.01$ and $\alpha_2 = \alpha_3 = 0.0004$. We drive 20 trials each of 2, 5 and 10 steps of 1 m and record the final pose against the reference, and turn on the spot by 90° 60 times.
2. α₃ from the along-track error. The sample variances divided by the distance give an estimate $\hat{\alpha}_3 = 0.00043$ (true 0.0004).
3. α₂ from the sideways error. The mean squares divided by $d(2d^2+1)/3$ give $\hat{\alpha}_2 = 0.00033$ (true 0.0004; the three distances alone give 0.00027, 0.00047 and 0.00025).
4. α₁ from the turns. The mean square turn error divided by $(\pi/2)^2$ gives $\hat{\alpha}_1 = 0.00978$ (true 0.01).
5. The error bars. Each estimate comes from about 60 values: a relative error of $\sqrt{2/60} = 18\%$. All three estimates are within 20% of the truth; the α₂ estimate is off by 17%, as expected. Repeating the α₂ experiment 60 times gives estimates with mean 0.00042 and spread 0.00014, against a predicted spread of 0.00013: the formula is unbiased and the error bar is right.
6. EM on the sensor data. Take the 2000 readings of the sensor lesson (truth: centre 2.012 m, noise 8 mm, 106 outliers, a share of 0.053), and start EM from a deliberately poor guess: centre at the plain mean (2.006), σ = 0.3 m, ε = 0.5. After 1 iteration: σ = 0.055 m, ε = 0.195; after 3: σ = 0.0083 m, ε = 0.0522; after 10 it has converged to centre 2.0119, σ = 0.0083, ε = 0.0526.
7. Validation. The lag-1 autocorrelation of the residuals is 0.017 (within ±0.046, so consistent with independent noise), and the share of readings within 1 and 2 fitted σ is 68.1% and 95.3%, the Gaussian values.

import math
from robotprob import Rng, sample_motion_odometry, gaussian_pdf, mean, variance

true_alphas = (0.01, 0.0004, 0.0004, 0.0004)
rng = Rng(2)
def drive(d):                                    # d steps of 1 m straight; returns the final pose
    p = [0.0, 0.0, 0.0]
    for _ in range(d):
        p = sample_motion_odometry(rng, p, (0.0, 1.0, 0.0), true_alphas)
    return p
def lateral_factor(d):                           # Var(sideways error) = alpha2 * d (2 d^2 + 1) / 3
    return d * (2 * d * d + 1) / 3.0

a3_terms, a2_terms = [], []
for d in (2, 5, 10):                             # 20 trials at each distance
    xs, ys = [], []
    for _ in range(20):
        p = drive(d)
        xs.append(p[0])
        ys.append(p[1])
    a3_terms.append(variance(xs) / d)            # Var(along-track error) = alpha3 * d
    a2_terms.append(mean([y * y for y in ys]) / lateral_factor(d))
print('alpha3: true', true_alphas[2], ' estimate', round(mean(a3_terms), 5))
print('alpha2: true', true_alphas[1], ' estimate', round(mean(a2_terms), 5), ' per distance:', [round(a, 5) for a in a2_terms])
turns = [sample_motion_odometry(rng, [0.0, 0.0, 0.0], (math.pi / 2, 0.0, 0.0), true_alphas)[2] - math.pi / 2 for _ in range(60)]
print('alpha1: true', true_alphas[0], ' estimate', round(mean([t * t for t in turns]) / (math.pi / 2) ** 2, 5))
print('expected relative standard error of an estimate from 60 values: +-', round(math.sqrt(2 / 60), 3))

rng = Rng(8)                                     # the sensor readings of the previous module
z = []
for _ in range(2000):
    out = rng.bernoulli(0.06)
    z.append(rng.uniform(0.2, 4.0) if out else 2.012 + rng.gauss(0.0, 0.008))

def em(z, mu, sigma, eps, lo, hi, iters):        # EM for: good Gaussian reading (prob 1 - eps) or a uniform outlier (prob eps)
    n = len(z)
    u = 1.0 / (hi - lo)
    history = []
    for _ in range(iters):
        r = []
        for v in z:                              # E step: responsibility of the Gaussian, by Bayes' rule
            g = (1 - eps) * gaussian_pdf(v, mu, sigma)
            r.append(g / (g + eps * u))
        sw = sum(r)                              # M step: weighted mean, weighted spread, outlier share
        mu = sum(ri * v for ri, v in zip(r, z)) / sw
        sigma = math.sqrt(sum(ri * (v - mu) ** 2 for ri, v in zip(r, z)) / sw)
        eps = 1.0 - sw / n
        history.append((mu, sigma, eps))
    return history

h = em(z, mean(z), 0.3, 0.5, 0.2, 4.0, 30)       # a deliberately poor starting guess
for k in (1, 3, 10, 30):
    mu, sg, ep = h[k - 1]
    print('EM iteration', str(k).rjust(2), ': centre', round(mu, 4), ' sigma', round(sg, 4), ' outlier share', round(ep, 4))
print('truth: centre 2.012, sigma 0.008, outlier share 106/2000 =', 106 / 2000)
mu, sg, ep = h[-1]
res = [v - mu for v in z if abs(v - mu) < 4 * sg]
r1 = sum(res[i] * res[i + 1] for i in range(len(res) - 1)) / sum(e * e for e in res)
print('lag-1 autocorrelation of the residuals', round(r1, 3), ' (about 0 for independent noise, +-', round(2 / math.sqrt(len(res)), 3), ')')
print('share of readings within 1 and 2 fitted sigma:', round(sum(1 for e in res if abs(e) < sg) / len(res), 3), round(sum(1 for e in res if abs(e) < 2 * sg) / len(res), 3), ' (68% and 95% expected)')

alpha3: true 0.0004  estimate 0.00043
alpha2: true 0.0004  estimate 0.00033  per distance: [0.00027, 0.00047, 0.00025]
alpha1: true 0.01  estimate 0.00978
expected relative standard error of an estimate from 60 values: +- 0.183
EM iteration  1 : centre 2.0111  sigma 0.0552  outlier share 0.1953
EM iteration  3 : centre 2.0119  sigma 0.0083  outlier share 0.0522
EM iteration 10 : centre 2.0119  sigma 0.0083  outlier share 0.0526
EM iteration 30 : centre 2.0119  sigma 0.0083  outlier share 0.0526
truth: centre 2.012, sigma 0.008, outlier share 106/2000 = 0.053
lag-1 autocorrelation of the residuals 0.017  (about 0 for independent noise, +- 0.046 )
share of readings within 1 and 2 fitted sigma: 0.681 0.953  (68% and 95% expected)

In robotics

A calibration protocol

For a differential-drive robot: (1) bias first: drive straight 10 m 10 times and turn 360° 10 times and compare with the reference; a mean error is a scale or wheel-base error to correct in the odometry, not noise. (2) Then the noise: with the biases removed, the residual spreads give $\alpha_3$ (along-track), $\alpha_2$ (sideways, from drives of several lengths) and $\alpha_1$ (turns on the spot). (3) Validate on new runs: coverage and, once a filter runs, the consistency tests of module 7. The UAV course describes the same discipline for inertial sensors using Allan variance in IMU noise and bias modelling.

Fitting the beam model

The four weights and σ_hit of the beam model of the last lesson are fitted by the same EM idea with four components instead of two: log a scan against a known map, compute for each beam the expected range, and let the responsibilities sort the residuals into hits, short readings, misses and random readings. Navigation stacks often ship with default weights; an afternoon of EM on logs from the real site is usually worth more than any tuning by eye.

Tuning versus identification

A common shortcut is to adjust the noise parameters until a filter's output looks smooth or its trajectory looks right. That fits the parameters to the filter's performance instead of to the sensor and hides modelling errors (an incomplete state, correlated noise) behind inflated numbers. Identification from reference data keeps them separate: if a filter with correctly identified noise is inconsistent, the model structure is wrong, and that is information worth having.

Cost of experiments

Calibration needs a reference, which may be a motion-capture room, a total station or a very careful tape measure. The √(2/n) law tells how much of that expense is needed: 30 trials per condition for ±25%, 200 for ±10%. Plan the conditions (distances, speeds, floors, payloads) first, because noise depends on them: odometry on carpet is not odometry on tile.

Parameters drift

Tyre wear, payload, battery voltage, temperature and floor all change the parameters. A calibrated robot is calibrated for a condition; recheck after any hardware change, and monitor the live innovations of the filter, which show when the assumed noise no longer matches reality.

Common misconceptions

Key takeaways

1. Each noise parameter is identified with an experiment that isolates it and a reference that knows the truth: straight drives for α₃ and α₂, turns on the spot for α₁, a fixed wall for a range sensor.
2. Divide the measured mean square error by its theoretical factor: α₃ = Var(along)/d, α₂ = E[y²]/(d(2d² + 1)/3), α₁ = E[θ²]/δ². Remove biases first; they are calibrated, not modelled.
3. A variance from n trials has a relative error √(2/n) (a standard deviation 1/√(2n)): 45% for 10 trials, 10% for 200. Always state the error bar.
4. For unlabelled outliers, EM alternates Bayes-rule responsibilities and weighted refitting. Start from robust estimates and keep σ from collapsing.
5. Validate on held-out data: coverage of ±1σ and ±2σ should be 68% and 95%, and residuals should be uncorrelated in time.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• S. Thrun, W. Burgard and D. Fox, Probabilistic Robotics, MIT Press, 2005, sections 5.3, 5.4 and 6.3 (learning the parameters of the motion and beam models, with the EM algorithm of table 6.2). Paid book.
• J. Borenstein and L. Feng, “Measurement and correction of systematic odometry errors in mobile robots”, IEEE Transactions on Robotics and Automation 12 (6), 1996. Paid journal article.
• T. D. Barfoot, State Estimation for Robotics, 2nd edition, Cambridge University Press, 2024 , chapter “Handling Nonidealities in Estimation” (bias, noise estimation). Free to read as the author’s PDF; the book is copyrighted.
• K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022 , chapter 8 (optimization, including the EM algorithm for mixtures). Free to read as the draft PDF, licence CC BY-NC-ND.
• C. Stachniss, “Observation Models” (video lecture) , including learning the sensor model. Free to watch.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Module 5 is complete: you have the two models, the structure that ties them together, and a way to identify their parameters. Module 6 combines them into the Bayes filter, the recursive estimator behind every localization method that follows.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T3: Range, Bearing and Beam Observation Models

  Write the likelihood of a range-bearing measurement of a landmark and of a range beam with hits, short readings, misses and random returns.

• Lab
  M5-L1: Probabilistic Odometry

  Dead reckoning puts Rover on the far side of a wall after two corners. Implement the odometry motion model, grow the uncertainty with distance, and identify the wheel noise from logged runs.

Other courses on this site treat neighbouring topics from their own angle:

• SLAM M1-T5: Odometry and Motion Models
• SLAM: Motion and Observation Models
• Robot Foundations 05: Where Am I?
• UAV M3-T7: IMU noise and bias modelling