M4-T1 — Expectation and Variance

Learning objectives

After this lesson, you should be able to:

• Compute the expectation and variance of discrete and continuous random variables, and say what each means physically.
• Use linearity of expectation and the rule that variances of independent quantities add, with proofs.
• Explain why random errors grow like the square root of the number of steps while bias grows in proportion to it, and compute the crossover.
• Distinguish E[g(X)] from g(E[X]), and explain why the sample variance divides by n − 1.

The intuition

Return to the clay of the first lessons: a unit of probability spread along an axis. The expectation (or mean) is where the clay would balance on a knife edge, its centre of mass. For the five camera frames of the Module 2 detector the bars balance at 4.0: on average four of five frames detect the marker. The expectation need not be a value that can occur, and it is a long-run average rather than a prediction: any single run gives 3, 4 or 5.

The variance measures how widely the clay is spread around that balance point. It is the average squared distance from the mean. Squaring has two virtues: it makes distances on both sides positive, and, unlike absolute values, it makes spreads add when independent quantities are combined, which the derivation below proves. The standard deviation is the square root of the variance, which brings it back to the unit of the quantity (a variance of centimetres squared would be hard to picture): for the five frames, 0.89 detections.

The right-hand panel is the single most important consequence in the lesson for a mobile robot. Suppose each metre driven adds an independent error with standard deviation 2 cm. After 100 m the error is not 100 × 2 cm = 200 cm but 2 cm × √100 = 20 cm: independent errors partly cancel, so the total grows only as the square root of the number of steps. A systematic error does not cancel. A bias of just 0.5 cm per metre adds to 50 cm after 100 m and dominates the random error after 16 m. This is why calibration of wheel diameters matters more than buying a quieter sensor, and why “odometry drift” is usually a bias problem.

Random errors add in quadrature, so they grow like √n. Systematic errors add up directly, so they grow like n. Distinguish them before deciding what to fix.

Definitions

Expectation $\mathbb{E}[X]$ or $\mu$

The probability-weighted average of the values of $X$: the centre of mass of the distribution. Also written $\langle X \rangle$ or $\bar{X}$.

Variance $\mathrm{Var}(X)$ or $\sigma^2$

The expected squared distance from the mean. Its unit is the unit of $X$ squared.

Standard deviation $\sigma$

The square root of the variance, in the unit of $X$. A typical size of a deviation from the mean.

Expectation of a function $\mathbb{E}[g(X)]$

The probability-weighted average of $g(x)$: first transform each value, then average.

Sample mean $\bar{x}$ and sample variance $s^2$

Estimates of the above computed from $n$ readings: the average, and the sum of squared deviations from the average divided by $n-1$.

The values for the distributions of the last module, worth knowing by heart:

Derivation: the rules for combining means and variances

Variance as a difference

1. $$\mathrm{Var}(X) = \mathbb{E}\bigl[X^2 - 2\mu X + \mu^2\bigr]$$

   Expand the square inside the expectation.

2. $$= \mathbb{E}[X^2] - 2\mu\,\mathbb{E}[X] + \mu^2$$

   Expectation is linear: it passes through sums and constants (proved next). $\mu$ is a constant.

3. $$= \mathbb{E}[X^2] - 2\mu^2 + \mu^2 = \mathbb{E}[X^2] - \mu^2$$

   Use $\mathbb{E}[X] = \mu$. The variance is the mean of the square minus the square of the mean, and so $\mathbb{E}[X^2] = \sigma^2 + \mu^2$ (always at least $\mu^2$).

Linearity of expectation

1. $$\mathbb{E}[aX + bY] = \sum_{x,y} (a x + b y)\, p(x, y)$$

   The expectation of a function of two variables, summed over the joint table.

2. $$= a\sum_x x \sum_y p(x, y) + b\sum_y y \sum_x p(x, y) = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$$

   Split the sum and marginalize (last module): $\sum_y p(x,y) = p(x)$. No assumption about independence was used, so the mean of a sum is always the sum of the means.

Sums of independent quantities

1. $$\mathrm{Var}(X + Y) = \mathbb{E}\bigl[(X - \mu_X + Y - \mu_Y)^2\bigr] = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2\,\mathbb{E}\bigl[(X - \mu_X)(Y - \mu_Y)\bigr]$$

   Expand the square of the sum of the two deviations. The last term is called the covariance of $X$ and $Y$ (next lesson).

2. $$\text{independent: } \mathbb{E}\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] = \mathbb{E}[X - \mu_X]\;\mathbb{E}[Y - \mu_Y] = 0 \cdot 0 = 0$$

   For independent variables the expectation of a product is the product of the expectations (the joint table factorizes), and each centred variable has mean zero.

3. $$\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y), \qquad \mathrm{Var}(aX) = a^2\,\mathrm{Var}(X)$$

   The second formula: $\mathbb{E}[(aX - a\mu)^2] = a^2\,\mathbb{E}[(X-\mu)^2]$.

4. $$S_n = X_1 + \dots + X_n \;\Rightarrow\; \mathrm{Var}(S_n) = n\sigma^2,\quad \sigma_{S_n} = \sigma\sqrt{n}, \quad \mathrm{Var}(\bar{X}) = \dfrac{\sigma^2}{n}$$

   For $n$ independent copies. The spread of the total grows as $\sqrt{n}$ and the spread of the average shrinks as $1/\sqrt{n}$. The second is the standard error of the Monte Carlo lesson; the first is the odometry drift above.

Why the sample variance divides by n − 1

The true variance is the average squared distance from the true mean, which is unknown. We use the sample mean instead, and that is a closer point to the data than the truth is.

1. $$\sum_{i}(x_i - \bar{x})^2 = \sum_{i}(x_i - \mu)^2 - n(\bar{x} - \mu)^2$$

   Write $x_i - \bar{x} = (x_i - \mu) - (\bar{x} - \mu)$, expand, and use $\sum_i (x_i - \mu) = n(\bar{x} - \mu)$. The sum around the sample mean is smaller than the sum around the true mean.

2. $$\mathbb{E}\Bigl[\sum_{i}(x_i - \bar{x})^2\Bigr] = n\sigma^2 - n\cdot\dfrac{\sigma^2}{n} = (n-1)\,\sigma^2$$

   Take expectations: each $(x_i-\mu)^2$ has mean $\sigma^2$, and $\mathbb{E}[(\bar{x}-\mu)^2] = \sigma^2/n$ from the previous derivation.

3. $$\mathbb{E}[s^2] = \dfrac{1}{n-1}\,(n-1)\sigma^2 = \sigma^2$$

   Dividing by $n-1$ makes the estimator unbiased. Dividing by $n$ would give $\tfrac{n-1}{n}\sigma^2$: 20% too small for five readings.

The sum of two Gaussians is Gaussian

The mean and variance of a sum follow from the rules above for any distributions. That the sum of independent Gaussians is again a Gaussian needs one more step, a convolution, and completing the square. For zero means, with variances $\sigma_1^2$ and $\sigma_2^2$:

1. $$f_{X+Y}(x) = \int f_X(t)\, f_Y(x - t)\,dt \;\propto\; \int \exp\!\Bigl(-\dfrac{t^2}{2\sigma_1^2} - \dfrac{(x - t)^2}{2\sigma_2^2}\Bigr)\,dt$$

   The density of a sum of independent variables is the convolution of the densities: every way of splitting $x$ into $t$ and $x - t$.

2. $$-\dfrac{t^2}{2\sigma_1^2} - \dfrac{(x - t)^2}{2\sigma_2^2} = -\dfrac{\bigl(t - x\sigma_1^2/(\sigma_1^2+\sigma_2^2)\bigr)^2}{2\,\sigma_1^2\sigma_2^2/(\sigma_1^2+\sigma_2^2)} - \dfrac{x^2}{2(\sigma_1^2 + \sigma_2^2)}$$

   Complete the square in $t$. The first term depends on $t$ and the second does not.

3. $$f_{X+Y}(x) \;\propto\; \exp\!\Bigl(-\dfrac{x^2}{2(\sigma_1^2 + \sigma_2^2)}\Bigr)$$

   The integral over $t$ of the first term is a Gaussian integral, a constant that does not depend on $x$. What remains is a Gaussian in $x$ with variance $\sigma_1^2 + \sigma_2^2$.

The mean of a sum is the sum of the means, always. The variance of a sum is the sum of the variances when the parts are independent. Averaging n independent readings divides the standard deviation by √n; adding n independent errors multiplies it by √n.

A worked example

Four calculations, all printed by the program below.

1. Mean and variance from a mass function. For the five-frame count with $p = 0.8$: $\mathbb{E}[X] = \sum k\,p(k) = 4.0$ and $\mathrm{Var}(X) = \sum (k - 4)^2 p(k) = 0.8$, matching the formulas $np = 4$ and $np(1-p) = 0.8$.
2. Odometry drift. Each metre adds an independent error with standard deviation 2 cm. After 10 m the standard deviation is $2\sqrt{10} = 6.32$ cm and after 100 m $2\sqrt{100} = 20$ cm (simulation: 6.37 and 19.91). A bias of 0.5 cm per metre gives a mean error of 5.0 cm and 50 cm (simulation: 5.02 and 49.78). The bias overtakes the noise where $0.5\,n = 2\sqrt{n}$, at $n = 16$ m.
3. A function of an average. A robot drives a fixed distance at a speed that varies uniformly between 0.5 and 1.5 m/s, average 1.0 m/s. The time per metre is $1/v$. Its expectation is $\int_{0.5}^{1.5} \dfrac{1}{v}\,dv = \ln 3 = 1.0986$ s per metre, not $1/\mathbb{E}[v] = 1$: the trip takes on average 10% longer than at the average speed. (Simulation: 1.101.)
4. The divisor n − 1. Draw five readings from a distribution with variance exactly 1, compute the sample variance, and average over 20 000 repetitions. Dividing by $n$ gives 0.801; dividing by $n - 1$ gives 1.005.
5. Sum of Gaussians. Adding independent Gaussians with standard deviations 3 and 4 gives 4.986 in simulation: $\sqrt{3^2 + 4^2} = 5$.

import math
from robotprob import Rng, mean, std, variance, binomial_pmf

pmf = {k: binomial_pmf(k, 5, 0.8) for k in range(6)}            # detections in 5 frames
m = sum(k * p for k, p in pmf.items())
v = sum((k - m) ** 2 * p for k, p in pmf.items())
print('binomial(5, 0.8): mean', round(m, 4), ' variance', round(v, 4), ' formulas', 5 * 0.8, round(5 * 0.8 * 0.2, 4))

rng = Rng(5)                                                    # odometry: 2 cm of independent noise per metre
def drift(n_steps, bias):
    return sum(bias + rng.gauss(0.0, 2.0) for _ in range(n_steps))
for n_steps in (10, 100):
    noise_only = [drift(n_steps, 0.0) for _ in range(4000)]
    with_bias = [drift(n_steps, 0.5) for _ in range(4000)]
    print(n_steps, 'm: noise only std', round(std(noise_only), 2), 'cm (theory', round(2 * math.sqrt(n_steps), 2), ')',
          ' with bias mean', round(mean(with_bias), 2), 'cm (theory', 0.5 * n_steps, ')')

speeds = [rng.uniform(0.5, 1.5) for _ in range(200000)]         # E[g(X)] is not g(E[X])
print('mean speed', round(mean(speeds), 3), ' mean of 1/speed', round(mean([1 / s for s in speeds]), 3), ' exact', round(math.log(3), 4))

def avg_var(ddof):                                              # why the sample variance divides by n - 1
    t = 0.0
    for _ in range(20000):
        t += variance([rng.gauss(0.0, 1.0) for _ in range(5)], ddof)
    return t / 20000
print('sample variance of 5 readings, true variance 1: divide by n ->', round(avg_var(0), 3), ' divide by n-1 ->', round(avg_var(1), 3))

s = [rng.gauss(0.0, 3.0) + rng.gauss(0.0, 4.0) for _ in range(100000)]
print('sum of N(0, 3^2) and N(0, 4^2): std', round(std(s), 3), ' (sqrt(9 + 16) = 5)')

binomial(5, 0.8): mean 4.0  variance 0.8  formulas 4.0 0.8
10 m: noise only std 6.37 cm (theory 6.32 )  with bias mean 5.02 cm (theory 5.0 )
100 m: noise only std 19.91 cm (theory 20.0 )  with bias mean 49.78 cm (theory 50.0 )
mean speed 0.999  mean of 1/speed 1.101  exact 1.0986
sample variance of 5 readings, true variance 1: divide by n -> 0.801  divide by n-1 -> 1.005
sum of N(0, 3^2) and N(0, 4^2): std 4.986  (sqrt(9 + 16) = 5)

In robotics

Error budgets: random drift versus bias

Dead-reckoning error after a distance $d$ has a random part that grows like $\sqrt{d}$ and a systematic part that grows like $d$. The estimate of the covariance that a localizer carries between corrections follows the √ law; a calibration error (a wheel radius off by 1%) shows up as a straight line of error that no covariance model describes. Plotting position error against distance on a log-log plot separates the two at a glance: a slope of ½ is noise, a slope of 1 is bias. The SLAM course shows the typical drift curves in Odometry and Motion Models.

Averaging and its limits

Averaging $n$ independent readings reduces the standard deviation by $\sqrt{n}$. Going from 1 cm to 1 mm takes 100 readings, and each extra digit costs a hundred times more. It does nothing for the bias or for readings that are not independent (M3-T3: Independence and Conditional Independence).

Nonlinear functions of an uncertain quantity

Time is distance over speed, energy is mass times speed squared, a range is the square root of a sum of squares. For a nonlinear function the average of the function is not the function of the average. In the example, a symmetrical spread of speeds makes the trip take 10% longer than at the average speed. A planner that uses the mean speed to predict arrival times is therefore optimistic. M4-T4: Propagating Uncertainty Through Transformations derives how to push a whole distribution through a nonlinear function.

Variance as a performance metric

The root-mean-square error of a trajectory, the spread of repeated trials, and the standard deviation of a controller's tracking error are all variances in disguise. The mean error tells you whether the robot is biased, the standard deviation whether it is repeatable, and both are needed: a robot can be unbiased and useless, or very repeatable and consistently wrong.

Software defaults differ

NumPy's var and std divide by n by default, while Matlab, R and most statistics packages divide by n − 1. For three readings the two standard deviations differ by 22%. State in your code which one you mean.

Common misconceptions

Key takeaways

1. The expectation is the balance point of the distribution, the variance the average squared distance from it, and the standard deviation its square root, in the unit of the quantity.
2. Expectation is linear, always. Variance adds only for independent quantities: Var(X + Y) = Var(X) + Var(Y), Var(aX) = a² Var(X).
3. n independent errors add to a standard deviation σ√n; their average has σ/√n. A systematic error adds up as n and overtakes the random part after a crossover distance.
4. E[g(X)] ≠ g(E[X]) for nonlinear g. The gap is the reason uncertain quantities cannot be pushed through nonlinear functions by their means alone.
5. The sample variance divides by n − 1 to be unbiased: dividing by n is too small by a factor (n − 1)/n. Check which one your software uses.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• J. Tsitsiklis and P. Jaillet, MIT RES.6-012 Introduction to Probability , lectures 5 to 7 (expectation and variance) and lecture 12 (sums of independent random variables). Free videos and notes, licence CC BY-NC-SA 4.0.
• J. Blitzstein and J. Hwang, Introduction to Probability, 2nd edition, CRC Press, 2019 , chapter 4 (expectation) and chapter 6 (moments). Free to read online; the book is copyrighted.
• T. D. Barfoot, State Estimation for Robotics, 2nd edition, Cambridge University Press, 2024 , chapter “Primer on Probability Theory”. Free to read as the author’s PDF; the book is copyrighted.
• C. Piech, Probability for Computer Scientists (Stanford CS109 course reader) , part 2 (expectation, variance). Free to read online.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The variance describes the uncertainty of one quantity. A robot’s pose has three at once, and they are not independent. The next lesson introduces the covariance that describes how uncertainties move together.

Related content

Nothing below is required to finish this lesson.

Other courses on this site treat neighbouring topics from their own angle:

• UAV M3-T2: Gaussian distributions and covariance
• SLAM M2-L3: Understand Pose Uncertainty
• Navigation M7-T3: Navigation Under Pose Uncertainty