M4-T2 — Covariance and Correlation

Learning objectives

After this lesson, you should be able to:

• Compute the covariance and the correlation coefficient of two quantities, and say what their sign and size mean physically.
• Write and read a covariance matrix, and compute the variance of a quantity along any direction from it.
• Show that correlation is a unit-free number between −1 and 1, and give an example of uncorrelated but dependent variables.
• Derive the number of independent readings that n equally correlated readings are worth.

The intuition

Rover drives 5 m along a diagonal path, 45° to the corridor walls. The wheels slip a little more or less in each run, so it stops short or long along the diagonal by about 5 cm, and the heading drifts so that it ends up to one side by about 1 cm. In the world frame the final error has an x part and a y part, and they are not independent. When Rover stops long, both x and y are too large; when it stops short, both are too small. The cloud of end points is stretched along the diagonal.

That is what covariance measures. It is the average of the product of the two errors, each measured from its own mean:

• If large x errors tend to come with large y errors, and small with small, then the product is positive more often than negative: positive covariance.
• If a large x error tends to come with a small y error, the covariance is negative.
• If they have no consistent relation, the positive and negative products cancel, and the covariance is zero.

Covariance has a unit (cm² here), so its size is hard to judge. Dividing by the two standard deviations gives the correlation coefficient ρ, a pure number between −1 and +1 that says how closely the cloud hugs a straight line: 0 for a round cloud, ±1 for points exactly on a line. In the example ρ = 0.92, nearly a line.

The covariance matrix

For two errors, the four numbers (variance of x, variance of y, and the covariance twice) are written as a 2 × 2 table, the covariance matrix. The diagonal holds the variances, the off-diagonal entries the covariance. For the diagonal drive it is [[13, 12], [12, 13]] cm². The large off-diagonal entry is the diagonal stretch. For three errors (x, y, heading) it is 3 × 3, and every filter in the second half of this course carries one.

Uncorrelated is not the same as independent

Independent variables always have zero covariance. The reverse fails, and the fourth panel shows why: if Rover circles a pillar, the position (cos θ, sin θ) is spread over a ring. The positive and negative products cancel by symmetry, so ρ = 0, yet knowing x tells you y up to a sign. Correlation detects linear relationships only.

Covariance describes the tilt and stretch of an error cloud. A round cloud means no relationship; a thin diagonal cloud means that knowing one error nearly tells you the other.

Definitions

Covariance $\mathrm{Cov}(X, Y)$ or $\sigma_{XY}$

The expected product of the deviations from the means. Unit: unit of $X$ times unit of $Y$.

Correlation $\rho$ or $\rho_{XY}$

The covariance divided by the product of the standard deviations. No unit, between −1 and +1.

Covariance matrix $\Sigma$

The matrix of all variances (on the diagonal) and covariances (off the diagonal) of a vector of random variables. It is symmetric, and positive semi-definite: no direction has a negative variance.

Uncorrelated

$\mathrm{Cov}(X,Y) = 0$. Weaker than independent.

Derivation: bounds, directions and the value of correlated readings

Why the correlation is between −1 and 1

Standardize both quantities: $U = (X - \mu_X)/\sigma_X$ and $V = (Y - \mu_Y)/\sigma_Y$ each have mean 0 and variance 1, and $\mathrm{Cov}(U, V) = \rho$.

1. $$\mathrm{Var}(U - sV) = \mathrm{Var}(U) + s^2\,\mathrm{Var}(V) - 2s\,\mathrm{Cov}(U, V) = 2 - 2s\rho$$

   Use the formula for the variance of a sum from the last lesson, with the covariance term kept. $s = \pm 1$.

2. $$2 - 2s\rho \ge 0 \quad\text{for } s = +1 \text{ and } s = -1 \;\;\Rightarrow\;\; -1 \le \rho \le 1$$

   A variance cannot be negative.

3. $$\rho = \pm 1 \;\Leftrightarrow\; \mathrm{Var}(U \mp V) = 0 \;\Leftrightarrow\; U = \pm V \text{ with probability 1}$$

   A variance of zero means the quantity never varies: $Y$ is an exact straight-line function of $X$.

Variance along a direction

1. $$\mathrm{Var}(aX + bY) = \mathbb{E}\bigl[(a(X - \mu_X) + b(Y - \mu_Y))^2\bigr]$$

   Definition, with the centred variables.

2. $$= a^2\,\mathbb{E}[(X-\mu_X)^2] + 2ab\,\mathbb{E}[(X-\mu_X)(Y-\mu_Y)] + b^2\,\mathbb{E}[(Y-\mu_Y)^2]$$

   Expand the square and use linearity of expectation.

3. $$= a^2\sigma_x^2 + 2ab\,\sigma_{xy} + b^2\sigma_y^2 = \mathbf{u}^{\top}\Sigma\,\mathbf{u}$$

   Recognise a quadratic form with $\mathbf{u} = (a, b)^{\top}$. For a unit vector it is the variance of the error along that direction.

How many independent readings are n correlated readings worth?

Let $n$ readings each have variance $\sigma^2$ and every pair have correlation $\rho$. Their average has:

1. $$\mathrm{Var}(\bar{X}) = \dfrac{1}{n^2}\Bigl[\sum_{i}\mathrm{Var}(X_i) + \sum_{i \ne j}\mathrm{Cov}(X_i, X_j)\Bigr]$$

   The variance of a sum is the sum of all entries of the covariance matrix, and $\bar{X} = \tfrac{1}{n}\sum X_i$ scales it by $1/n^2$.

2. $$= \dfrac{1}{n^2}\Bigl[n\sigma^2 + n(n-1)\,\rho\,\sigma^2\Bigr] = \dfrac{\sigma^2}{n}\bigl(1 + (n-1)\rho\bigr)$$

   There are $n$ variances and $n(n-1)$ ordered pairs, each with covariance $\rho\sigma^2$.

3. $$\dfrac{\sigma^2}{n_{\text{eff}}} = \mathrm{Var}(\bar{X}) \;\;\Rightarrow\;\; n_{\text{eff}} = \dfrac{n}{1 + (n-1)\rho}$$

   Define the equivalent number of independent readings as the number that would give the same variance of the average. For $\rho = 0$ it is $n$; for $\rho = 1$ it is 1; and as $n \to \infty$ it approaches $1/\rho$ however many readings you take.

Correlation limits what averaging can achieve. Equally correlated readings never improve beyond σ²ρ, however many are taken: the correlated part of the error is the same in every reading and does not average away.

A worked example

Rover drives along a 45° diagonal. The error along the path has standard deviation 5 cm, the error across it 1 cm, and the two are independent. Rotating into the world frame gives the x and y errors.

1. Covariance matrix by calculation. The variances along and across are $\sigma_a^2 = 25$ and $\sigma_c^2 = 1$ cm². With $\cos 45° = \sin 45° = 1/\sqrt{2}$: $\sigma_x^2 = \sigma_y^2 = (25 + 1)/2 = 13$ and $\sigma_{xy} = (25 - 1)/2 = 12$. So $\Sigma = \begin{pmatrix} 13 & 12 \\ 12 & 13 \end{pmatrix}$ cm² and $\rho = 12/13 = 0.923$.
2. By simulation. 20 000 simulated end points give [[13.10, 12.09], [12.09, 13.10]] and a correlation of 0.923.
3. Variance along a direction. Along the path $\mathbf{u} = (0.707, 0.707)$: $\mathbf{u}^{\top}\Sigma\mathbf{u} = 0.5(13) + 2(0.5)(12) + 0.5(13) = 25$ cm², a standard deviation of 5 cm. Across the path: 1 cm². Looking only at the x variance (13 cm², standard deviation 3.6 cm) would hide both.
4. A ring. If the end point is spread uniformly around a pillar, $(\cos\theta, \sin\theta)$, the simulated correlation is −0.003: zero, within sampling error, although $x^2 + y^2 = 1$ always.
5. Correlated readings. Five readings with pairwise correlation 0.8: the variance of their average is $\tfrac{1}{5}(1 + 4 \times 0.8) = 0.84$ (simulation 0.827), the same as $5/(1 + 3.2) = 1.19$ independent readings. The five readings are worth little more than one.

import math
from robotprob import Rng, cov_matrix, correlation, variance

rng = Rng(3)
c, s = math.cos(math.pi / 4), math.sin(math.pi / 4)
pts = []
for _ in range(20000):
    a = rng.gauss(0.0, 5.0)                      # error along the direction of travel (cm)
    b = rng.gauss(0.0, 1.0)                      # error across it (cm)
    pts.append([c * a - s * b, s * a + c * b])   # rotated into the world frame
S = cov_matrix(pts)
xs = [p[0] for p in pts]
ys = [p[1] for p in pts]
print('covariance matrix  =', [[round(v, 2) for v in row] for row in S], ' theory [[13, 12], [12, 13]]')
print('correlation        =', round(correlation(xs, ys), 3), ' theory', round(12 / 13, 3))

def var_dir(S, u):                               # u^T S u: variance along the unit vector u
    return u[0] * u[0] * S[0][0] + 2 * u[0] * u[1] * S[0][1] + u[1] * u[1] * S[1][1]
print('variance along, across =', round(var_dir(S, [c, s]), 2), round(var_dir(S, [-s, c]), 2), ' theory 25, 1')

ring = []                                        # uncorrelated, but dependent
for _ in range(20000):
    t = rng.uniform(0.0, 2 * math.pi)
    ring.append([math.cos(t), math.sin(t)])
print('ring: correlation', round(correlation([p[0] for p in ring], [p[1] for p in ring]), 3), ' but x^2 + y^2 is always', round(ring[0][0] ** 2 + ring[0][1] ** 2, 3))

n, rho = 5, 0.8                                  # equally correlated readings
means = []
for _ in range(20000):
    common = rng.gauss(0.0, 1.0)                 # the part of the error shared by all five readings
    xs5 = [math.sqrt(rho) * common + math.sqrt(1 - rho) * rng.gauss(0.0, 1.0) for _ in range(n)]
    means.append(sum(xs5) / n)
print('variance of the mean of 5 readings with rho = 0.8:', round(variance(means), 3), ' formula', round((1 + (n - 1) * rho) / n, 3), ' equivalent independent readings', round(n / (1 + (n - 1) * rho), 2))

covariance matrix  = [[13.1, 12.09], [12.09, 13.1]]  theory [[13, 12], [12, 13]]
correlation        = 0.923  theory 0.923
variance along, across = 25.2 1.01  theory 25, 1
ring: correlation -0.003  but x^2 + y^2 is always 1.0
variance of the mean of 5 readings with rho = 0.8: 0.827  formula 0.84  equivalent independent readings 1.19

In robotics

Pose uncertainty is a matrix

A robot's position estimate is reported with a covariance matrix, not a radius. A long, thin cloud means the robot knows its position well in one direction and poorly in the other: in a long corridor, lidar scan matching pins the position across the corridor and says little along it. A planner that uses only the larger of the two standard deviations is too cautious across the corridor and one that uses the smaller is reckless along it. ROS 2 pose messages carry this matrix (the 6 × 6 covariance field of geometry_msgs/PoseWithCovariance), and the Navigation course shows how to turn it into a safety margin along the direction to the nearest wall, exactly the formula $\mathbf{u}^{\top}\Sigma\mathbf{u}$ of this lesson, in Navigation Under Pose Uncertainty. The SLAM view is in Understand Pose Uncertainty.

Heading error produces correlated position errors

A small heading error $\delta\theta$ displaces the position sideways by about $d\,\delta\theta$ after a distance $d$. Because the same heading error enters the position error of every later step, the position errors at different times are correlated, and the pose covariance develops off-diagonal terms between heading and position. A filter that ignored them (a diagonal covariance) would be overconfident about position after any turn. M5-T2: Odometry and Velocity Motion Models derives this for the odometry model.

Why a Kalman filter needs off-diagonal terms

A measurement of x alone improves the estimate of y when their errors are correlated, and by how much is exactly the correlation (module 4, M4-T5: Conditioning and Fusing Gaussians, and module 7). This is how a filter that only measures position can estimate velocity, and a camera that sees one landmark can correct a heading.

Correlated noise in sensors

Successive readings from an IMU, GNSS receiver or camera frame share slowly varying errors (bias, multipath, lighting), so their errors are correlated in time. The equivalent number of independent readings, $n/(1 + (n-1)\rho)$, is the right number to use when averaging them, and the reason why 100 Hz data does not make a 100 times better estimate. The UAV course discusses this for inertial sensors in IMU noise and bias modelling.

Reading a covariance from data

To measure the correlation of two errors, collect repeated trials and compute the sample covariance matrix. Check it is not nearly singular (a correlation of 0.99999 is a sign that two of your states are the same quantity under two names), and remember that the sample covariance is itself uncertain when there are few samples.

Common misconceptions

Key takeaways

1. Covariance is the average product of two deviations from their means: positive when the errors rise together, negative when they oppose, zero when they have no linear relation.
2. The correlation coefficient ρ = σxy/(σxσy) is unit-free, between −1 and +1, and is ±1 exactly for a straight-line relationship.
3. The covariance matrix collects the variances (diagonal) and covariances (off diagonal). The variance along a unit vector u is uᵀΣu.
4. Independent implies uncorrelated; uncorrelated does not imply independent (a ring has ρ = 0).
5. n readings with pairwise correlation ρ are worth n/(1 + (n − 1)ρ) independent readings, and no more than about 1/ρ however large n is.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• J. Tsitsiklis and P. Jaillet, MIT RES.6-012 Introduction to Probability , lecture 12 (sum of independent random variables, covariance and correlation). Free videos and notes, licence CC BY-NC-SA 4.0.
• T. D. Barfoot, State Estimation for Robotics, 2nd edition, Cambridge University Press, 2024 , chapter “Primer on Probability Theory” (covariance and the multivariate Gaussian). Free to read as the author’s PDF; the book is copyrighted.
• J. Blitzstein and J. Hwang, Introduction to Probability, 2nd edition, CRC Press, 2019 , chapter 7 (covariance and correlation). Free to read online; the book is copyrighted.
• ROS 2 geometry_msgs/PoseWithCovariance message definition , which fixes the order of the six covariance rows (x, y, z and rotations about them). Free documentation.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

With the covariance matrix we can describe a cloud of errors in any number of dimensions. The next lesson puts it inside the Gaussian density, and turns the matrix into the ellipse that robotics engineers draw.

Related content

Nothing below is required to finish this lesson.

• Before
  M4-T1: Expectation and Variance

  Compute the expectation and variance of discrete and continuous variables, use linearity of expectation, and relate the sample mean to the true mean.

• Lab
  M4-L1: Robot Uncertainty Visualizer

  The position estimate comes with a covariance that nobody on the team can read. Compute it from samples, draw its ellipse, and push it through a rotation and a range-bearing conversion.

Other courses on this site treat neighbouring topics from their own angle:

• UAV M3-T2: Gaussian distributions and covariance
• SLAM M2-L3: Understand Pose Uncertainty
• Navigation M7-T3: Navigation Under Pose Uncertainty