M5-T3 — Geometric Interpretation of Residuals

Learning objectives

After this lesson, you should be able to:

• Explain the least-squares prediction A x̂ as the projection of b onto the column space of A.
• State and justify that the residual is orthogonal to every column of A, and what that implies (for example residuals summing to zero when A contains a column of ones).
• Use Pythagoras to split ‖b‖² into explained and unexplained parts.
• Use the orthogonality as a unit test for a fit and the residual fraction as a quality measure.

Introduction and motivation

It is easy to run a fit; it is harder to know whether to trust it. The geometry gives you tests that do not depend on the particular data. The residual must be orthogonal to every column of the design matrix, which means (for a line fit with an intercept) that the residuals sum to zero and are uncorrelated with the inputs. When a solver returns an answer violating this, there is a bug or an inaccuracy. And the Pythagorean split of the data into fitted and leftover parts gives the standard quality measure for a fit.

The intuition

Collect all the data you could possibly predict: every vector of the form A x for some choice of unknowns. They form a flat set, the column space of A (a plane, for two unknowns). The measured data b is a point generally off that plane, because of noise. The best prediction is the point on the plane nearest to b, and the nearest point is found by dropping a perpendicular.

That perpendicularity is the entire content of the normal equations: “the residual is orthogonal to every column of A” is the condition for the minimum.

Geometric and physical interpretation

Geometric view. The m-dimensional data space contains the n-dimensional column space of A. The least-squares prediction is the orthogonal projection of b onto it, and the residual lies in the orthogonal complement, the left null space N(Aᵀ) of the earlier module. b = (fit part) + (error part), with the two parts perpendicular. Physical view. The fit part is the portion of the measurements the model can explain; the error part is everything it cannot. If the model is right and the noise is random, the error part is only noise. If the error part is large or structured, the model is missing something (a bias, a nonlinearity, another sensor effect).

The mathematics

p = A x̂

the least-squares prediction (projection of b).

r = b − A x̂

the residual (here defined as data minus prediction).

C(A)

the column space: all possible predictions.

N(Aᵀ)

the left null space: the orthogonal complement of C(A).

Orthogonality of the residual

Aᵀ r = Aᵀ (b − A x̂) = 0          i.e.   ai · r = 0 for every column ai of A

What it means. For each column (each feature of the model) the residual has no component along it. If a column is all ones, this says the residuals sum to zero; if a column holds the x-values, the residuals are uncorrelated with x.

Pythagoras

‖b‖² = ‖A x̂‖² + ‖r‖²          R² = ‖A x̂‖² / ‖b‖² = 1 − ‖r‖²/‖b‖²

What it means. The squared length of the data equals the squared length of the fitted part plus that of the residual. The fraction explained is a standard quality number (after subtracting the mean when appropriate).

Closest point

‖b − A x‖² ≥ ‖b − A x̂‖²   for every x          (equality only at x̂ when A has independent columns)

What it means. Any other prediction is farther from the data, by Pythagoras: the extra distance is the length of the segment between that prediction and the projection.

Least squares = projection of b onto C(A). The residual is perpendicular to every column of A, and ‖b‖² = ‖Ax̂‖² + ‖r‖².

Step-by-step derivation: orthogonality from the minimum, and Pythagoras

The minimum forces orthogonality. Suppose x̂ minimises ‖b − Ax‖². Perturb it by a direction d and a small number t: the cost of x̂ + t d is

‖b − A x̂ − t A d‖² = ‖r − t A d‖² = ‖r‖² − 2 t (r · A d) + t² ‖A d‖²

What it means. Expanding the square gives a quadratic in t. At t = 0 (the claimed minimum) it must not decrease, so its slope there must be zero.

The slope at t = 0 is −2 r·(Ad). For a minimum this must vanish for every direction d, so r · (Ad) = 0 for all d. Taking d = ej gives r · aj = 0 for each column: Aᵀ r = 0. Conversely, if Aᵀr = 0 then the quadratic's slope vanishes and, since its t² coefficient is non-negative, t = 0 is a minimum.

Pythagoras. Since b = Ax̂ + r with r ⟂ C(A) and Ax̂ ∈ C(A), we get ‖b‖² = ‖Ax̂‖² + 2(Ax̂)·r + ‖r‖² = ‖Ax̂‖² + ‖r‖², because the cross term is zero.

Residuals sum to zero. If the first column of A is all ones, a₁ = (1, …, 1), then a₁·r = Σ ri = 0: the residuals have zero mean.

The residual lies in the left null space. Aᵀr = 0 says exactly r ∈ N(Aᵀ), so the data split as b = (a vector of C(A)) + (a vector of N(Aᵀ)), the splitting of R^m into two perpendicular subspaces from the module on subspaces.

A worked example: checking orthogonality and Pythagoras on a line fit

Take the fit of the earlier lesson: A = [[1, 1], [2, 1], [3, 1]], b = (1, 2, 2), x̂ = (0.5, 0.6667).

1. Prediction. p = A x̂ = (1.1667, 1.6667, 2.1667).
2. Residual (data minus prediction). r = b − p = (−0.1667, 0.3333, −0.1667).
3. Orthogonal to the column of ones: −0.1667 + 0.3333 − 0.1667 = 0 ✓ (the residuals sum to zero).
4. Orthogonal to the column of x-values (1, 2, 3): −0.1667 + 0.6667 − 0.5 = 0 ✓.
5. Pythagoras. ‖b‖² = 1 + 4 + 4 = 9, ‖p‖² = 1.3611 + 2.7778 + 4.6944 = 8.8333, ‖r‖² = 0.0278 + 0.1111 + 0.0278 = 0.1667, and 8.8333 + 0.1667 = 9 ✓. The line explains 98.1% of the squared data length.
6. Another point is worse. The prediction from the line y = x is (1, 2, 3), and its residual (0, 0, −1) has squared length 1, larger than 0.1667, and is not orthogonal to the ones column (sum −1).

Robotics application

A unit test for any fit. After solving, compute Aᵀ r and check it is zero to machine precision (relative to ‖A‖‖r‖). It catches a wrong sign convention, a bad weighting, a stale matrix and an inaccurate solver (see conditioning, later). Most fitting bugs make this test fail.

Diagnosing sensor models. If the residuals are small and look like noise, the model is adequate. If they are structured (a drift with time, larger at long range), the model misses an effect, for instance a scale error that grows with distance. Calibration workflows plot residuals against each input for this reason.

LiDAR plane fitting. The residual of each point is its distance to the fitted plane; the residual vector is perpendicular to the columns of the design matrix. The standard deviation of the residuals estimates the sensor noise and is used to set the ground-segmentation threshold. See the SLAM lesson point-cloud preprocessing.

Explained fraction. R² and similar numbers summarise how much of the measured variation a model explains; in robotics they are used to compare, for example, an odometry model with and without a slip term.

Engineering implications and limitations

Practical rules

• Always compute and look at the residual vector. It is free after the solve and tells you more than the parameters do.
• Use Aᵀr = 0 as an assertion. Normalise by ‖A‖‖r‖ and expect about 10⁻¹⁵ for a well-conditioned problem. A value near 10⁻⁶ indicates an ill-conditioned problem or an inaccurate method.
• Zero-mean residuals only if there is an intercept. Without a column of ones in A (a fit forced through the origin) the residuals need not sum to zero.
• A small residual does not prove a correct model. A model with many parameters fits noise and leaves a small residual. Check the residuals on data not used in the fit.
• Mind the scale of R². Whether R² uses the raw data or the data with its mean removed changes its meaning; state which.

Limits

The perpendicularity statements are about the ordinary dot product and plain least squares. With weights (Module 5, later) the residual is orthogonal in the weighted sense (AᵀW r = 0), and in total least squares the errors are measured perpendicular to the model instead.

Common misconceptions

Engineering takeaways

1. The least-squares prediction A x̂ is the orthogonal projection of b onto the column space of A.
2. The residual r = b − A x̂ is orthogonal to every column of A (Aᵀr = 0) and so lies in the left null space.
3. Pythagoras: ‖b‖² = ‖A x̂‖² + ‖r‖²; the fraction explained is a quality measure.
4. With a column of ones in A the residuals sum to zero.
5. Test any solver output with Aᵀr ≈ 0; inspect the residuals for structure, outliers and model errors.

References and further reading

Primary papers and authoritative documentation.

• MIT 18.06SC Linear Algebra (OCW): lecture on projections onto subspaces and least squares
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): the chapter on least squares and the orthogonality principle; the site did not load when we last checked
• Mathematics for Machine Learning: the chapter on orthogonal projection and linear regression
• 3Blue1Brown, Essence of Linear Algebra: the geometry of projections

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The perpendicularity condition Aᵀr = 0 can be written out and solved. That gives the normal equations, the classical way to compute a least-squares fit, together with an honest account of when it is accurate.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T2: Least-Squares Formulation

  Set up a fit as minimising the sum of squared residuals and write it in matrix form.