M9-T6 — Whitening Transformations

Learning objectives

After this lesson, you should be able to:

• Construct a whitening matrix W with WΣWᵀ = I using the eigen-decomposition (symmetric square root) or the Cholesky factor.
• Explain that whitening is not unique (rotations) and which choice suits which purpose.
• Use whitening to turn generalised least squares into ordinary least squares and to compute Mahalanobis distances.
• Apply it to residuals in a SLAM or calibration problem with mixed units.

Introduction and motivation

A pose-graph edge mixes a translation error in metres with a rotation error in radians, with a covariance that couples them. A calibration problem fits a model to range readings with noise that grows with distance. Giving every residual the same weight would let the loudest sensor dominate. Whitening each residual by the square root of its information matrix puts them all on equal footing, and it is exactly how solvers such as Ceres, g2o and GTSAM implement weighted least squares.

The intuition

A correlated cloud is a round cloud that has been stretched along some axes and turned. Whitening undoes that: rotate to the axes of the ellipse (the eigenvectors), divide each axis by its standard deviation (the square root of the eigenvalue), and (optionally) rotate back. After whitening, the cloud is round with unit radius; distances in the new coordinates are Mahalanobis distances in the old ones.

Geometric and physical interpretation

Geometric view. With Σ = QΛQᵀ the whitening W = Λ^(−1/2)Qᵀ (PCA whitening) first rotates into the eigenbasis then rescales each axis to unit variance; W = QΛ^(−1/2)Qᵀ = Σ^(−1/2) (ZCA whitening) rotates back afterwards so the result resembles the original axes as closely as possible; W = L⁻¹ (Cholesky whitening) is lower triangular, so each whitened variable depends only on the original variables up to its index. All three give WΣWᵀ = I and differ by an orthogonal matrix. Physical view. A whitened residual has no preferred direction and no units: every component is “how many standard deviations”.

The mathematics

Σ

a positive definite covariance matrix.

W

a whitening matrix: W Σ Wᵀ = I.

Σ = QΛQᵀ, Σ = LLᵀ

the eigen- and Cholesky decompositions.

Ω = Σ⁻¹

the information matrix; WᵀW = Ω.

Three whitening matrices

WPCA = Λ^(−1/2) Qᵀ        WZCA = Σ^(−1/2) = Q Λ^(−1/2) Qᵀ        Wchol = L⁻¹

What it means. All satisfy WΣWᵀ = I and WᵀW = Σ⁻¹. They differ by an orthogonal matrix: W₂ = U W₁ with UᵀU = I. Choose Cholesky for speed, PCA for decorrelated components, ZCA to stay close to the original coordinates.

Whitened data and distance

w = W (x − μ)  ⇒  Cov(w) = I,        d² = (x − μ)ᵀΣ⁻¹(x − μ) = ‖w‖²

What it means. Because WᵀW = Σ⁻¹, the squared norm of the whitened deviation equals the squared Mahalanobis distance.

Generalised least squares

min (Ax − b)ᵀ Σ⁻¹ (Ax − b)  ⇔  min ‖ (WA) x − (Wb) ‖²,   x̂ = (AᵀΣ⁻¹A)⁻¹ AᵀΣ⁻¹ b

What it means. Whiten the equations by W and solve ordinary least squares (Module 5). The normal equations of the whitened problem are exactly the weighted normal equations.

W Σ Wᵀ = I, WᵀW = Σ⁻¹. Whitening turns weighted least squares into ordinary least squares and Mahalanobis distance into Euclidean distance. W is unique up to rotation.

Step-by-step derivation: the three whiteners and weighted least squares

1. PCA whitening. With Σ = QΛQᵀ and W = Λ^(−1/2)Qᵀ: WΣWᵀ = Λ^(−1/2)QᵀQΛQᵀQΛ^(−1/2) = Λ^(−1/2)ΛΛ^(−1/2) = I.
2. ZCA whitening. W = QΛ^(−1/2)Qᵀ is the symmetric inverse square root of Σ. Then WΣW = QΛ^(−1/2)Λ Λ^(−1/2)Qᵀ = I. It equals PCA whitening followed by the rotation Q.
3. Cholesky whitening. From Σ = LLᵀ: L⁻¹ΣL⁻ᵀ = L⁻¹LLᵀL⁻ᵀ = I. It costs one triangular factorisation and later triangular solves (no eigen-decomposition).
4. Non-uniqueness. If W₁ΣW₁ᵀ = I and U is orthogonal, then (UW₁)Σ(UW₁)ᵀ = UW₁ΣW₁ᵀUᵀ = UUᵀ = I. Conversely any two whiteners differ by an orthogonal matrix (compare W₂W₁⁻¹).
5. Information matrix. WᵀW = W₁ᵀUᵀUW₁ = W₁ᵀW₁ = Σ⁻¹ for every choice: the information matrix is the shared outer product, so the weighted cost does not depend on which whitener is used.
6. Weighted least squares. Consider b = Ax + n with n ~ N(0, Σ). Multiply by W: Wb = WAx + Wn with Wn ~ N(0, I): a standard problem with white noise. Its least-squares solution is (AᵀWᵀWA)⁻¹AᵀWᵀWb = (AᵀΣ⁻¹A)⁻¹AᵀΣ⁻¹b, the generalised least-squares estimate. Its covariance is (AᵀΣ⁻¹A)⁻¹.
7. Maximum likelihood. The Gaussian log-likelihood is −½(Ax − b)ᵀΣ⁻¹(Ax − b) + const, so maximum likelihood is exactly weighted least squares, and equally least squares on the whitened residuals.
8. In nonlinear solvers. Each residual rᵢ(x) with covariance Σᵢ is multiplied by Lᵢ⁻¹ (stored as the “square-root information” matrix) before it enters the sum of squares; the Jacobian is multiplied by the same factor.

Data whitening versus noise whitening. Whitening a data set uses its sample covariance (a preprocessing step in learning and perception); whitening a residual uses the known or modelled noise covariance (an estimation step). The algebra is identical.

Conditioning. Whitening can be ill-conditioned: if Σ has a tiny eigenvalue, Λ^(−1/2) has a huge entry (a nearly deterministic direction becomes enormously weighted). Regularise by adding a small multiple of the identity or clipping small eigenvalues before whitening.

A worked example: three whiteners and a correlated estimate

• Covariance. Σ = [[2, 1], [1, 2]] with eigenvalues 3 and 1 and eigenvectors (1, 1)/√2, (1, −1)/√2.
• ZCA. W = QΛ^(−1/2)Qᵀ = [[0.789, −0.211], [−0.211, 0.789]]; check WΣWᵀ = I ✓ (to rounding).
• Cholesky. L = [[1.414, 0], [0.707, 1.225]], W = L⁻¹ = [[0.707, 0], [−0.408, 0.816]], again WΣWᵀ = I. It differs from the ZCA matrix by a rotation (and is triangular).
• PCA. W = Λ^(−1/2)Qᵀ = [[−0.707, 0.707], [0.408, 0.408]] (signs depend on the library) with WΣWᵀ = I.
• Whitened distance. For v = (1, 0): d² = vᵀΣ⁻¹v = 2/3 = 0.667 and the ZCA-whitened vector is Wv = (0.789, −0.211) with ‖Wv‖² = 0.622 + 0.045 = 0.667 ✓.
• Weighted estimation. Two readings of one scalar, z = (10, 12), with noise covariance Σ = [[1, 0.5], [0.5, 4]] (correlated). Generalised least squares x̂ = (1ᵀΣ⁻¹1)⁻¹ 1ᵀΣ⁻¹ z = 10.25 with variance (1ᵀΣ⁻¹1)⁻¹ = 0.9375. Whitening with the Cholesky factor gives A = L⁻¹1 = (1, 0.258), b = L⁻¹z = (10, 3.615) and the ordinary least-squares answer (AᵀA)⁻¹Aᵀb = 10.25 ✓, the same.
• Compare with ignoring correlation. With Σ = diag(1, 4) the estimate is 10.4 with variance 0.8: the correlation changes both the estimate (10.25) and its honesty (0.9375 instead of 0.8: less information than the independent model claims).

Robotics application

SLAM and pose graphs. Each edge residual is whitened with the Cholesky factor of its information matrix before optimisation; see pose graphs and nonlinear least squares, and the weighted least-squares lesson of Module 5.

LiDAR registration. Point-to-plane or covariance-weighted ICP whitens each correspondence with its own covariance; see ICP fundamentals and scan registration.

UAV and sensor fusion. Square-root Kalman filters propagate the Cholesky factor of P instead of P itself for numerical stability; innovation whitening gives the standard tests of filter consistency; see the error-state Kalman filter and extrinsic calibration.

Perception and learning. Whitening the descriptors before clustering or nearest-neighbour search makes Euclidean distance meaningful; ZCA is common in vision preprocessing. Manipulators weight position and orientation errors by whitening in task space (Module 8).

Engineering implications and limitations

Practical rules

• Prefer Cholesky for solvers. It is the cheapest and its triangular structure keeps sparsity; use PCA or ZCA when you need decorrelated or minimally changed coordinates.
• Store square-root information. Keeping L⁻¹ (or the upper-triangular factor of Σ⁻¹) per residual avoids repeated factorisations and squared condition numbers.
• Regularise before whitening. Clip or inflate tiny eigenvalues; whitening a near-singular Σ amplifies noise enormously.
• Whiten with the right covariance. The noise covariance (known) for residuals; the data covariance (estimated) for preprocessing; do not mix them up.
• Scale mixed units through whitening. A residual of metres and radians becomes dimensionless; the optimiser then treats all components consistently.
• Check the result. Compute the sample covariance of whitened residuals: it should be close to the identity. Large departures indicate a wrong or stale Σ.

Limits

Whitening assumes the covariance is correct and, for a Gaussian interpretation, that the noise is Gaussian. It is a linear operation and cannot fix heavy tails or outliers (robust losses are applied after whitening, on the whitened residual norm).

Common misconceptions

Engineering takeaways

1. A whitening matrix satisfies WΣWᵀ = I and WᵀW = Σ⁻¹; options are Λ^(−1/2)Qᵀ (PCA), Σ^(−1/2) (ZCA) and L⁻¹ (Cholesky), all equal up to rotation.
2. After whitening, the Euclidean norm equals the Mahalanobis distance.
3. Weighted (generalised) least squares is ordinary least squares on whitened equations: x̂ = (AᵀΣ⁻¹A)⁻¹AᵀΣ⁻¹b with covariance (AᵀΣ⁻¹A)⁻¹.
4. Solvers store square-root information and whiten each residual and Jacobian block; correlations change both the estimate and its variance.
5. Regularise tiny eigenvalues, verify the whitened covariance is the identity, and use robust losses after whitening.

References and further reading

Primary papers and authoritative documentation.

• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): weighted and regularised least squares, covariance and whitening; the site did not load when we last checked
• Barfoot, State Estimation for Robotics (author-hosted first edition, free to read): chapter 3-4, maximum likelihood and weighted least squares with Gaussian noise
• Mathematics for Machine Learning: chapter 9, linear regression with Gaussian noise (maximum likelihood)
• UCSD ECE276A: Sensing and Estimation in Robotics: square-root information and whitening in pose optimisation

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

We now have all the ingredients: Gaussians, covariance propagation, Mahalanobis gating and whitening. The last lesson of the module assembles them into the Kalman filter, a few matrix equations that predict and correct an estimate and its covariance.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T5: Mahalanobis Distance

  Measure distance in units of uncertainty and use it to gate measurements.

• Before
  M6-T3: Symmetric Matrices

  State the spectral theorem and use the orthogonal eigenbasis of a symmetric matrix.