M2-T3 — The Gaussian Distribution

Learning objectives

After this lesson, you should be able to:

• Read the mean and standard deviation of a Gaussian as a centre and a typical error, and sketch the curve from them.
• Use the 68-95-99.7 rule, z-scores and the standard normal table (or gaussian_cdf) to compute probabilities and tolerances.
• Derive the Gaussian normalization constant and the Gaussian shape from independence and rotational symmetry.
• Explain why the sum of many small independent errors is nearly Gaussian, and name three situations where a Gaussian is a poor model.

The intuition

Repeat the same measurement many times and plot the results. The values bunch around a centre, thin out symmetrically on both sides, and almost never lie far away. The smooth curve that describes this is the Gaussian, or normal, distribution. Two numbers are enough to draw it.

• The mean $\mu$ is the centre: where the curve peaks. A sensor whose mean error is not zero has a bias: it is wrong in the same direction every time.
• The standard deviation $\sigma$ is the width: the typical distance of a reading from the centre. It has the unit of the measurement, so a range sensor has a $\sigma$ in centimetres. A smaller $\sigma$ gives a taller, narrower curve; the area is always 1.

The right-hand panel is the most useful fact in this lesson. It lets you read a sensor's data sheet. If a range sensor has $\sigma = 2$ cm, then about two readings in three are within 2 cm of the truth, about 19 in 20 are within 4 cm, and a reading more than 6 cm off is a one-in-370 event. Note the wording: typical distance does not mean maximum distance.

Why a bell?

The bell shape is not an arbitrary choice. Most measurement errors are made of many tiny, independent causes: thermal noise in an electronic component, small vibrations, rounding, slight air turbulence, the jitter of a clock. Each contributes a little, in a random direction, and the total is their sum. The figure below shows what happens when you add independent errors that are individually not bell-shaped at all. One uniform error is flat. The sum of two is a triangle. The sum of three is already rounded, and the sum of twelve lies almost on top of the Gaussian curve.

That behaviour is the central limit theorem: the sum of many independent small errors is approximately Gaussian, whatever the shape of the individual errors. It explains why the Gaussian is a good default, and the conditions in its statement (many, small, independent) tell you when to distrust it.

There is a second reason, a practical one. The Gaussian is the one distribution family for which the operations of a filter keep the answer inside the family. Add two independent Gaussian errors and the total is Gaussian. Multiply a Gaussian error by a constant and it is still Gaussian. Fuse two Gaussian beliefs about the same thing and the result is Gaussian. Because of this, a belief can be stored as two numbers, a mean and a variance, instead of a whole curve, and that is the Kalman filter (module 7).

Remember the three numbers 68, 95 and 99.7, with 1, 2 and 3 standard deviations. They turn a standard deviation on a data sheet into a statement about how often a reading is wrong by how much.

Definitions

Gaussian $\mathcal{N}(\mu, \sigma^2)$

The distribution with mean $\mu$ and variance $\sigma^2$. We write $X \sim \mathcal{N}(\mu, \sigma^2)$ for “$X$ is Gaussian with these parameters”. The second argument is the variance, the square of the standard deviation; some software expects the standard deviation instead, a classic source of errors.

Mean $\mu$

The centre of the curve and its long-run average.

Standard deviation $\sigma$

The width: the typical distance from the centre. Same unit as $X$.

Variance $\sigma^2$

The square of the standard deviation. Unit: the unit of $X$ squared. Variances add for independent errors; standard deviations do not.

Standard normal $Z$

The Gaussian with $\mu = 0$ and $\sigma = 1$. Its cumulative distribution is written $\Phi(z)$.

z-score

$z = (x - \mu)/\sigma$: how many standard deviations $x$ lies from the centre. A z-score is a pure number and does not depend on the unit.

Properties used constantly

The proofs of the sum rule and of the product rule are in M4-T1: Expectation and Variance and M4-T5: Conditioning and Fusing Gaussians.

Derivation: where the constant and the bell come from

Why the factor $1/\sqrt{2\pi}$

The area under $e^{-x^2/2}$ must be computed so that the density integrates to 1. There is no antiderivative in elementary functions, but the area itself has a classical closed form, found by squaring it and changing to polar coordinates.

1. $$I = \int_{-\infty}^{\infty} e^{-x^2/2}\,dx \;\;\Rightarrow\;\; I^2 = \int_{-\infty}^{\infty}\!\int_{-\infty}^{\infty} e^{-(x^2 + y^2)/2}\,dx\,dy$$

   Square the integral and write it as a double integral over the whole plane, using a different name for the second variable.

2. $$I^2 = \int_0^{2\pi}\!\int_0^{\infty} e^{-r^2/2}\, r\,dr\,d\theta$$

   Switch to polar coordinates: $x^2 + y^2 = r^2$ and the area element is $r\,dr\,d\theta$. The extra factor $r$ is what makes the integral doable.

3. $$\int_0^{\infty} e^{-r^2/2}\, r\,dr = \Bigl[-e^{-r^2/2}\Bigr]_0^{\infty} = 1 \;\;\Rightarrow\;\; I^2 = 2\pi$$

   The inner integral has the antiderivative $-e^{-r^2/2}$. The angle integral is just $2\pi$.

4. $$I = \sqrt{2\pi} \;\;\Rightarrow\;\; \int \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2}\,dz = 1$$

   Divide by $I$ to normalize the standard normal. With the unit-change rule of the last lesson, $X = \mu + \sigma Z$ gives the density $\tfrac{1}{\sigma}\varphi\bigl(\tfrac{x-\mu}{\sigma}\bigr)$, the formula above.

Why the bell: no preferred direction

Rover drops a part on a table at a target point. The landing error has a horizontal part $x$ and a vertical part $y$. Make two assumptions that are easy to state physically, and the Gaussian is the only answer.

1. $$\text{density of the point } (x, y) = f(x)\, f(y)$$

   The two components are independent and have the same shape $f$ (the robot is no better at one axis than the other).

2. $$f(x)\, f(y) = g(x^2 + y^2)$$

   There is no preferred direction: the density depends only on the distance $r^2 = x^2 + y^2$ from the target, for some function $g$. A rotated table gives the same statistics.

3. $$h(x) = \ln\dfrac{f(x)}{f(0)} \;\;\Rightarrow\;\; h(x) + h(y) = h\bigl(\sqrt{x^2 + y^2}\bigr)$$

   Put $y = 0$ to get $g(x^2) = f(x)f(0)$, substitute that back into the first line to eliminate $g$, divide by $f(0)^2$ and take logarithms.

4. $$k(u) = h(\sqrt{u}) \;\;\Rightarrow\;\; k(x^2) + k(y^2) = k(x^2 + y^2),\quad\text{so } k(u + v) = k(u) + k(v)$$

   A function whose value at a sum is the sum of its values. For a continuous function this forces $k(u) = -\alpha u$ for a constant $\alpha$ (the only continuous solutions are straight lines through the origin; the sign is negative so that the area is finite).

5. $$f(x) = f(0)\, e^{-\alpha x^2}$$

   Undo the substitutions: $h(x) = k(x^2) = -\alpha x^2$. This is a Gaussian with $\sigma^2 = 1/(2\alpha)$, and normalization fixes $f(0)$.

A two-dimensional error cloud with independent axes and no preferred direction must be Gaussian. That is why the round, symmetric cloud of position errors in the Kalman filter is a Gaussian, and why a cloud that is stretched along a particular direction needs a covariance matrix (module 4) instead of a single σ.

A worked example

A range sensor has a Gaussian error with $\mu = 0$ and $\sigma = 2$ cm. Five questions about it, all answered by the program below.

1. Tolerance. How often is a reading within ±3 cm of the truth? $z = 3/2 = 1.5$, so $P = \Phi(1.5) - \Phi(-1.5) = 0.8664$: about 87%.
2. Specification. A new application needs 99% of the readings within ±5 cm. For 99% two-sided the z-score is 2.576, so $\sigma \le 5/2.576 = 1.94$ cm. The sensor must be slightly better than the one we have.
3. Averaging. The mean of 16 independent readings has $\sigma/\sqrt{16} = 0.5$ cm. The probability that this average is within ±1 cm is $z = 1/0.5 = 2$, so 0.9545. (The 16 readings must be independent; the correlation warning of the binomial lesson applies again.)
4. The bell from nothing. Add 12 independent uniform numbers on $[-0.5, 0.5]$. Each has variance $1/12$, so the sum has variance 1 and standard deviation 1. Simulating 20 000 such sums gives mean −0.002, standard deviation 0.999, and shares within 1, 2 and 3 of 0.6807, 0.9553 and 0.9978, against the Gaussian's 0.6827, 0.9545 and 0.9973.
5. Three sigma is routine. A scanning lidar delivers 1000 beams at 10 Hz, 10 000 readings a second. A fraction 0.0027 lies beyond 3σ even when nothing is wrong: 27 beams every second. Beyond 4σ it is still 0.63 beams a second. “More than three standard deviations” is therefore a poor definition of a faulty reading on its own.

from robotprob import Rng, gaussian_cdf, gaussian_ppf, mean, std

sigma = 0.02                                   # 2 cm, in metres
print('P(|e| <= 3 cm)          =', round(gaussian_cdf(0.03, 0.0, sigma) - gaussian_cdf(-0.03, 0.0, sigma), 4))

z = gaussian_ppf(0.995)                        # 99% two-sided leaves 0.5% in each tail
print('z for 99% two-sided     =', round(z, 3), ' sigma needed =', round(5 / z, 2), 'cm')

n = 16
s_mean = sigma / n ** 0.5                      # the average of n independent readings
print('sigma of the mean of', n, '  =', s_mean)
print('P(|mean error| <= 1 cm) =', round(gaussian_cdf(0.01, 0.0, s_mean) - gaussian_cdf(-0.01, 0.0, s_mean), 4))

rng = Rng(11)                                  # a sum of 12 uniform numbers is almost Gaussian
s = [sum(rng.uniform(-0.5, 0.5) for _ in range(12)) for _ in range(20000)]
print('sum of 12 uniforms: mean', round(mean(s), 3), 'std', round(std(s), 3))
for k in (1, 2, 3):
    print('within', k, 'sigma:', round(sum(1 for v in s if abs(v) <= k) / len(s), 4), ' Gaussian:', round(gaussian_cdf(k) - gaussian_cdf(-k), 4))

rate = 1000 * 10                               # beams per second
for k in (3, 4):
    q = 2 * (1 - gaussian_cdf(k))
    print(k, 'sigma: fraction', round(q, 6), '->', round(q * rate, 2), 'beams per second')

P(|e| <= 3 cm)          = 0.8664
z for 99% two-sided     = 2.576  sigma needed = 1.94 cm
sigma of the mean of 16   = 0.005
P(|mean error| <= 1 cm) = 0.9545
sum of 12 uniforms: mean -0.002 std 0.999
within 1 sigma: 0.6807  Gaussian: 0.6827
within 2 sigma: 0.9553  Gaussian: 0.9545
within 3 sigma: 0.9978  Gaussian: 0.9973
3 sigma: fraction 0.0027 -> 27.0 beams per second
4 sigma: fraction 6.3e-05 -> 0.63 beams per second

In robotics

Reading a data sheet

Data sheets quote noise in several ways, and the first job is to find out which. “Accuracy ±3 cm” may be a guaranteed bound, a 1σ value or a 95% interval. “Noise density 0.01 °/s/√Hz” for a gyroscope means a standard deviation that grows with the measurement bandwidth. Converting to a common σ before combining sensors is basic hygiene; the UAV course shows it for inertial sensors in IMU noise and bias modelling.

Combining independent errors

A ground robot's position error at the end of a corridor is the sum of a heading error and a distance error. Each is a Gaussian, so their combination is another, with variance equal to the sum. Square, add, and take the root: this is how an error budget is computed. Errors of 3 cm, 4 cm and 12 cm give 13 cm, and shrinking the smallest ones barely helps: improve the biggest first.

Setting thresholds and gates

A fixed threshold on a measurement residual, “reject if the difference between measured and predicted range exceeds 3σ”, is the Gaussian rule in practice. The Kalman filter gates with the Mahalanobis distance, the multi-dimensional version of the z-score (M4-T3: The Multivariate Gaussian and Covariance Ellipses, M8-T2: EKF Localization with Landmarks). As the beam example showed, a 3σ gate rejects 0.27% of perfectly good measurements; whether that is acceptable is an engineering decision.

Where a Gaussian is a poor model

• Heavy tails and outliers. A lidar beam that hits a person walking through the scan, a GPS multipath jump or a camera mismatch produces errors far beyond any Gaussian tail. Real sensor error is often a Gaussian core plus a small share of outliers; the next lesson models this.
• Bounded quantities. A range cannot be negative and a battery level cannot exceed 100%. A Gaussian puts a little probability on impossible values; when the mean is many σ from the boundary the effect is negligible, and when it is not (a sensor reading near zero range) it matters.
• Several modes. A robot in a corridor with three identical doors has a belief with three peaks. A single Gaussian cannot represent it, which is why module 6 begins with the histogram filter and module 9 with particles.
• Angles. A heading is on a circle, so a Gaussian on the real line is wrong near ±180°. Module 12 treats uncertainty on angles properly.

The SLAM and Perception courses use the same normal distribution; the SLAM view of it is in Probability and Uncertainty.

Common misconceptions

Key takeaways

1. A Gaussian is fixed by its mean (the centre, the place a bias moves) and its standard deviation (a typical error, in the unit of the measurement).
2. About 68% of values lie within 1σ, 95% within 2σ (1.96 for exactly 95%) and 99.7% within 3σ. A z-score counts standard deviations and has no unit.
3. Independent Gaussian errors add in variance, so standard deviations combine in quadrature. The average of n independent readings has standard deviation σ/√n.
4. Sums of many small independent errors are nearly Gaussian (the central limit theorem). A symmetric, independent two-dimensional error cloud must be Gaussian.
5. The Gaussian is a model. It fails for outliers, bounded quantities, several modes and angles, and 3σ readings are routine when there are thousands of readings a second.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• J. Tsitsiklis and P. Jaillet, MIT RES.6-012 Introduction to Probability , lectures 8 to 10 (continuous random variables, including the normal) and lecture 19 (the central limit theorem). Free videos and notes, licence CC BY-NC-SA 4.0.
• J. Blitzstein and J. Hwang, Introduction to Probability, 2nd edition, CRC Press, 2019 , chapters 5 and 10. Free to read online; the book is copyrighted.
• J. F. W. Herschel, review of Quetelet’s Letters on the Theory of Probabilities, Edinburgh Review, 1850, and J. C. Maxwell, “Illustrations of the dynamical theory of gases”, Philosophical Magazine, 1860: the two original symmetry arguments for the Gaussian. Paid journal articles.
• C. Stachniss, “Probability Primer for Probabilistic Robotics” (video lecture) . Free to watch.
• S. Thrun, W. Burgard and D. Fox, Probabilistic Robotics, MIT Press, 2005, section 2.2 (the normal distribution) and chapter 3. Paid book.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A real sensor is rarely a clean Gaussian. The next lesson separates what is in a typical sensor’s error (bias, noise, quantization and outliers) and builds models for each.

Related content

Nothing below is required to finish this lesson.

• Before
  M2-T2: Continuous Random Variables, Densities and the CDF

  Explain why a density is not a probability, and compute probabilities of intervals from a density or a cumulative distribution.

• Lab
  M2-L1: Sensor Noise Explorer

  A range sensor reads a wall at a fixed distance and the readings scatter. Build the histogram, estimate bias and spread, and find the outliers a Gaussian cannot explain.

Other courses on this site treat neighbouring topics from their own angle:

• UAV M3-T2: Gaussian distributions and covariance
• Perception P1-T5: Noise, Uncertainty, Occlusion and Ambiguity