M5-T6 — Weighted Least Squares

Learning objectives

After this lesson, you should be able to:

• Define weighted least squares: minimise Σ w_i r_i² = (Ax − b)ᵀW(Ax − b).
• Derive the weighted normal equations AᵀWA x = AᵀWb, and show it is ordinary least squares on rescaled data.
• Relate the weights to measurement variances, W = Σ⁻¹, and compute the fused estimate and its variance.
• Explain the effect of a mis-scaled or extreme weight.

Introduction and motivation

Sensor fusion is weighted least squares. If one range sensor has a 2 cm error and another a 20 cm error, giving them equal votes wastes the better one. If a pose-graph edge comes from a wheel odometer and another from a loop-closure match, they must be weighted by their uncertainty, or the optimiser will trust the wrong one. The weights are not arbitrary tuning knobs: when they are the inverse variances, the result is the best possible linear estimate.

The intuition

In the plain cost every residual counts equally. Now give residual i a weight wi and minimise Σ wi ri². A large weight makes a residual expensive, so the fit bends toward that point; a small weight lets the fit largely ignore it. Think of springs of different stiffness attached to the data points: the line settles where the stiff springs have their way.

Geometric and physical interpretation

Geometric view. Weights change the notion of distance in the data space: instead of the ordinary length, we measure with rᵀWr, which stretches some directions. Least squares is still a projection, but an orthogonal projection in this weighted geometry: the residual is orthogonal to the columns in the sense AᵀW r = 0. Physical view. A residual divided by its standard deviation is a “z-score”: how many sigmas away the model is from the measurement. Weighting by 1/σ² means every residual is judged in units of its own noise, which is the fair way to combine unlike sensors.

The mathematics

W

a diagonal (or, in general, symmetric positive definite) weight matrix; diagonal entries w_i > 0.

σ_i

the standard deviation of measurement i.

Σ

the covariance matrix of the measurements; for independent measurements it is diag(σ_i²).

The weighted problem

minimise  (A x − b)ᵀ W (A x − b) = Σ wi ri²        ⇒        (AᵀW A) x̂ = AᵀW b

What it means. The same derivation as before, with W inserted. The weighted Gram matrix AᵀWA and weighted right-hand side AᵀWb replace AᵀA and Aᵀb.

Reduction to the unweighted problem

W = SᵀS,  S = diag(√wi):      minimise ‖ S A x − S b ‖²

What it means. Multiplying each row of A and b by the square root of its weight turns weighted least squares into ordinary least squares. All the numerical advice (QR instead of the normal equations) carries over.

The right weights

W = Σ⁻¹   (wi = 1/σi² for independent measurements)        covariance of x̂ = (AᵀΣ⁻¹A)⁻¹

What it means. With these weights the estimate has the smallest possible variance among linear unbiased estimators (Gauss-Markov), and the matrix AᵀΣ⁻¹A, the information matrix, gives its covariance when inverted.

Weighted least squares: (AᵀWA)x̂ = AᵀWb. Choose W = Σ⁻¹, and the result is the best linear estimate with covariance (AᵀΣ⁻¹A)⁻¹.

Step-by-step derivation: the weighted normal equations and the fused variance

The weighted normal equations. The cost is f(x) = (Ax − b)ᵀW(Ax − b) with W symmetric positive definite.

1. Expand: f = xᵀ(AᵀWA)x − 2bᵀWAx + bᵀWb.
2. Gradient: ∇f = 2AᵀWAx − 2AᵀWb.
3. Set it to zero: AᵀWA x̂ = AᵀWb. Because AᵀWA is positive definite for independent columns, this is the unique minimum.

The same as ordinary least squares on rescaled rows. Write W = SᵀS (for a diagonal W, S = diag(√wi)). Then (Ax − b)ᵀW(Ax − b) = ‖S(Ax − b)‖² = ‖(SA)x − Sb‖², and the ordinary normal equations for SA and Sb are AᵀSᵀSAx = AᵀSᵀSb, i.e. the weighted equations.

The scalar case. Fusing two measurements z₁, z₂ of one unknown with weights w₁, w₂: A = [1, 1]ᵀ, so AᵀWA = w₁ + w₂ and AᵀWb = w₁z₁ + w₂z₂, giving x̂ = (w₁z₁ + w₂z₂)/(w₁ + w₂).

Why inverse variances. Suppose zi = x + ei with independent errors of variance σi². For a weighted average x̂ = Σ cizi with Σ ci = 1 (so it is unbiased), the variance is Σ ci²σi². Minimising this under the constraint (Lagrange multipliers) gives ci ∝ 1/σi². So the weights 1/σi² are the best, and the variance of the result is 1/Σ(1/σi²): information (inverse variance) adds.

A worked example: weighted line fit and two-sensor fusion

A weighted line fit. Fit y = mx + c to (1, 1), (2, 2), (3, 2), but trust the third point four times as much (weights 1, 1, 4): W = diag(1, 1, 4).

1. Weighted Gram matrix. AᵀWA = [[1·1 + 1·4 + 4·9, 1 + 2 + 12], [15, 1 + 1 + 4]] = [[41, 15], [15, 6]]. AᵀWb = (1·1·1 + 1·2·2 + 4·3·2, 1 + 2 + 8) = (29, 11).
2. Solve. The determinant is 41·6 − 15·15 = 21. m = (29·6 − 15·11)/21 = 9/21 = 0.4286 and c = (41·11 − 15·29)/21 = 16/21 = 0.7619.
3. Compare with unweighted (0.5, 0.6667): the weighted line predicts 0.4286·3 + 0.7619 = 2.0476 at x = 3, closer to the trusted y = 2 than the unweighted prediction 2.1667.

Two sensors. A distance is measured as 10.0 m by a sensor with σ = 2 m and as 10.6 m by a sensor with σ = 1 m. Weights: w₁ = 1/4 = 0.25 and w₂ = 1.

1. x̂ = (0.25·10.0 + 1·10.6)/(0.25 + 1) = 13.1/1.25 = 10.48 m: much closer to the precise sensor.
2. Variance. 1/(0.25 + 1) = 0.8 m², so the fused standard deviation is √0.8 = 0.894 m, better than either sensor alone (1 m and 2 m).

Robotics application

Sensor fusion. Combining a GPS fix (σ = 2 m) with a LiDAR localisation (σ = 0.1 m) with these weights makes the LiDAR dominate when it is available and lets the GPS take over when it is not. This is also the update step of a Kalman filter (Module 9); see Gaussian distributions and covariance and the error-state Kalman filter.

Pose-graph and bundle-adjustment optimisation. Each constraint (odometry edge, loop closure, landmark observation) enters the cost weighted by its inverse covariance (the “information matrix”). The SLAM track's pose graph lesson shows these weights, and robust optimisation describes what to do when a weight should not be trusted blindly.

Range-dependent noise. A LiDAR's error grows with distance, so far points should get smaller weights in a plane fit, or a few distant, noisy points will bend the plane.

Heterogeneous residuals. A residual vector combining positions (metres) and angles (radians) is meaningless without weights; dividing each by its noise standard deviation puts them on a common scale.

Engineering implications and limitations

Practical rules

• Use 1/σ² from the sensor specification or from measured noise. Do not invent weights to make a fit look better. If you do not know the noise, estimate it from repeated measurements at rest.
• Only relative weights matter for the estimate. Multiplying all weights by 10 changes nothing in x̂ (it changes the reported covariance), so overall scale is a convention.
• Extreme weights ruin conditioning. A weight ratio of 10¹⁰ makes the problem numerically nearly singular. Treat “exactly known” as a constraint, or cap the ratio.
• Apply the weights by scaling the rows and then use QR, rather than forming AᵀWA, to keep the numerical advantages of the previous lesson.
• An overconfident weight is worse than an honest one. Too-small variances make the estimator trust a bad sensor and ignore good ones. Check consistency: the weighted residuals should have unit size on average.

Limits

The statements about optimality assume independent (or correctly modelled correlated) errors, unbiased sensors and a correct model. A systematic bias is not averaged away by weighting, and an outlier with the correct weight is still an outlier: robust costs are needed there.

Common misconceptions

Engineering takeaways

1. Weighted least squares minimises Σ w_i r_i² = (Ax − b)ᵀW(Ax − b) and solves AᵀWA x̂ = AᵀWb.
2. It equals ordinary least squares on rows scaled by √w_i.
3. The right weights are inverse variances (W = Σ⁻¹); then x̂ is the best linear unbiased estimate with covariance (AᵀΣ⁻¹A)⁻¹.
4. Fusing two sensors gives x̂ = (w₁z₁ + w₂z₂)/(w₁ + w₂) with variance 1/(w₁ + w₂): information adds.
5. Pose graphs, Kalman filters and range-dependent fits are all weighted least squares.

References and further reading

Primary papers and authoritative documentation.

• UCSD ECE276A: Sensing and Estimation in Robotics: maximum likelihood estimation with Gaussian noise: weighted least squares
• Barfoot, State Estimation for Robotics (author-hosted first edition, free to read): the batch estimation problem and information matrices
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): the chapter on least squares data fitting, including weighted versions; the site did not load when we last checked
• MIT 18.06SC Linear Algebra (OCW): lecture on least squares with different noise levels

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Sometimes the data do not pin down the unknowns, or the fit is so sensitive that noise dominates. Regularisation adds a gentle preference that makes such problems well behaved, at a price you can measure.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T5: QR Decomposition

  Solve least squares through QR and explain why it is more accurate than the normal equations.

• Before
  M5-T4: Normal Equations

  Derive and solve the normal equations, and state when they are valid and when they lose accuracy.