M4-T3 — Orthogonal Projections

Learning objectives

After this lesson, you should be able to:

• Compute the projection of a vector onto a line and onto a subspace spanned by the columns of A.
• Derive P = A(AᵀA)⁻¹Aᵀ from the requirement that the residual is perpendicular to the subspace.
• Prove that the projection is the closest point and that P is symmetric and idempotent (P² = P).
• Apply projection to distance from a plane, ground points and sliding along a surface.

Introduction and motivation

A LiDAR return rarely lies exactly on the ground plane you have fitted. A commanded velocity may point slightly into a wall. A measurement of a robot position may violate a known constraint. In every case, the sensible correction is to move to the nearest consistent point, and the nearest point in a flat set is an orthogonal projection. The same geometry underlies least squares (Module 5), which is why the argument here is worth following in detail.

The intuition

Stand a stick upright above a tabletop and shine a light straight down: the shadow of its tip is the point of the table nearest to the tip. That shadow is the orthogonal projection of the tip onto the table. Two facts follow from the picture: the shadow is the closest point of the table to the tip, and the line from the tip to its shadow meets the table at a right angle.

Geometric and physical interpretation

Geometric view. The projection of b onto a subspace is the unique point p of the subspace such that b − p is perpendicular to the whole subspace. By Pythagoras, any other point of the subspace is farther from b, so p is the nearest. Physical view. The distance of a LiDAR point from a fitted ground plane is the length of the residual, and its foot on the plane is where the point “would be” if the ground were perfectly flat. A velocity command projected onto a wall's tangent line is the velocity that slides along the wall.

The mathematics

b

the vector to be projected.

A

a matrix whose columns span the subspace (with independent columns).

p

the projection of b onto the column space of A.

r

the residual b − p.

P

the projection matrix: p = P b.

Onto a line

p = ( (a · b) / (a · a) ) a          (line through a)

What it means. The coefficient (a·b)/(a·a) is how many copies of a the shadow of b contains. For a unit vector a this is the dot-product lesson formula.

Onto a subspace

p = A x̂,     AᵀA x̂ = Aᵀ b     ⇒     p = A (AᵀA)⁻¹ Aᵀ b = P b

What it means. The coefficients x̂ are chosen so the residual is perpendicular to every column of A. They solve the normal equations, which Module 5 uses for least squares. The matrix P performs the projection in one multiplication.

Properties of P

P² = P          Pᵀ = P          (I − P) projects onto the orthogonal complement
‖b‖² = ‖p‖² + ‖r‖²

What it means. Projecting twice changes nothing (once you are on the subspace you stay). P is symmetric. The leftover b − p is the projection onto the perpendicular space, and the squared lengths add by Pythagoras.

Projection: find the point p in the subspace with b − p perpendicular to the subspace. It is the closest point, and it is given by P = A(AᵀA)⁻¹Aᵀ.

Step-by-step derivation: the projection matrix and the closest-point property

Finding p. We look for p = A x̂ (a combination of the columns) such that the residual r = b − A x̂ is perpendicular to every column of A.

1. Perpendicular to every column means each dot product with a column is zero, which is the single matrix statement Aᵀ r = 0.
2. Substitute r = b − A x̂: Aᵀ(b − A x̂) = 0, so AᵀA x̂ = Aᵀ b (the normal equations).
3. If the columns of A are independent, AᵀA is invertible (shown in Module 5), so x̂ = (AᵀA)⁻¹Aᵀb.
4. Therefore p = A x̂ = A(AᵀA)⁻¹Aᵀ b, which defines P = A(AᵀA)⁻¹Aᵀ.

Properties. P² = A(AᵀA)⁻¹AᵀA(AᵀA)⁻¹Aᵀ = A(AᵀA)⁻¹Aᵀ = P, since AᵀA(AᵀA)⁻¹ = I. Pᵀ = (Aᵀ)ᵀ((AᵀA)⁻¹)ᵀAᵀ = A(AᵀA)⁻¹Aᵀ = P since AᵀA is symmetric so its inverse is as well.

The projection is the closest point. Let q be any other point of the subspace. Then b − q = (b − p) + (p − q), where b − p = r is perpendicular to the subspace and p − q lies in it. By Pythagoras:

‖b − q‖² = ‖b − p‖² + ‖p − q‖² ≥ ‖b − p‖²

What it means. The distance to q is the distance to p plus a non-negative extra, with equality only when q = p. So no point of the subspace is closer to b than p.

The line case. With a single column a: AᵀA = a·a, Aᵀb = a·b, so x̂ = (a·b)/(a·a) and p = x̂ a, the line formula.

A worked example: projecting onto a plane

Let the plane be spanned by a₁ = (1, 0, 1) and a₂ = (0, 1, 1), so A = [[1, 0], [0, 1], [1, 1]], and let b = (1, 2, 4).

1. Normal equations. AᵀA = [[2, 1], [1, 2]] and Aᵀb = (1 + 4, 2 + 4) = (5, 6).
2. Solve [[2, 1], [1, 2]] x̂ = (5, 6): the determinant is 3, so x̂ = (4/3, 7/3).
3. Projection. p = (4/3) a₁ + (7/3) a₂ = (4/3, 7/3, 11/3) ≈ (1.333, 2.333, 3.667).
4. Residual. r = b − p = (−1/3, −1/3, 1/3). Check: r·a₁ = −1/3 + 1/3 = 0 and r·a₂ = −1/3 + 1/3 = 0 ✓ perpendicular to both. Its length is ‖r‖ = √(1/3) ≈ 0.577: the distance from b to the plane.
5. Pythagoras. ‖b‖² = 21, ‖p‖² = 20.667, ‖r‖² = 0.333; they add to 21. ✓
6. Onto a line. Projecting (3, 4) onto the line through (1, 0) gives (3, 0); onto (1, 1) gives (7/2)(1, 1) = (3.5, 3.5).

Robotics application

Distance to a fitted plane. For a plane through a point with unit normal n̂, the residual of projecting a point onto the plane is the component along n̂, of length |δ| with δ = n̂·(x − p₀), as in the cross-product lesson. Ground segmentation classifies a LiDAR point as ground if |δ| is below a few centimetres.

Sliding along surfaces. Removing the component of a velocity along the wall normal is projection onto the wall's tangent line or plane; collision-avoidance layers do this every cycle.

Enforcing constraints. A state estimate that violates a linear constraint (for example the components of an orientation vector must sum to a fixed value) can be corrected by projecting it onto the constraint set; this is the simplest form of constrained estimation.

Least squares. Fitting a line or plane to noisy points is a projection of the data vector onto the space of exactly fitting data; Module 5 builds on this lesson, and the Navigation track's costmap and ICP lessons apply the same geometry (see ICP fundamentals).

Engineering implications and limitations

Practical rules

• Do not form P explicitly. For a large A, P is a dense m × m matrix. Compute the coefficients x̂ by a solver (QR, Module 5) and then p = A x̂.
• Do not form (AᵀA)⁻¹. Solve the normal equations, or better still use QR; the explicit inverse is slow and loses accuracy, especially when columns are nearly dependent.
• Orthonormal columns make it trivial. If the columns of Q are orthonormal, P = QQᵀ and p = Σ (qi·b) qi: no solve at all.
• Check the residual. After projecting, Aᵀ r should be at machine precision relative to ‖A‖‖b‖. A large value means the columns were nearly dependent or the solve was inaccurate.
• Independent columns are required. If the columns of A are dependent, AᵀA is singular; use the pseudoinverse (Module 6) instead.

Limits

Orthogonal projection gives the nearest point in the ordinary (Euclidean) sense. If errors in different directions matter differently, the right notion of “nearest” is a weighted projection, taken up with weighted least squares in Module 5.

Common misconceptions

Engineering takeaways

1. The projection p of b onto a subspace is the point of the subspace with b − p perpendicular to the subspace.
2. It is the closest point (by Pythagoras) and p = A x̂ with AᵀA x̂ = Aᵀb.
3. P = A(AᵀA)⁻¹Aᵀ satisfies P² = P and Pᵀ = P; I − P projects onto the perpendicular space.
4. With orthonormal columns Q, P = QQᵀ and no solve is needed.
5. Plane distances, sliding motion and constraint enforcement are projections, and least squares is built on them.

References and further reading

Primary papers and authoritative documentation.

• MIT 18.06SC Linear Algebra (OCW): lecture on projections onto subspaces and least squares
• 3Blue1Brown, Essence of Linear Algebra: dot products and the geometry of projection
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): the chapter on least squares, which uses this geometry; the site did not load when we last checked
• Mathematics for Machine Learning: the chapter on analytic geometry: orthogonal projections

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Projection needs a good set of directions to project onto. The Gram-Schmidt process builds a perpendicular set from any independent vectors, step by step.

Related content

Nothing below is required to finish this lesson.

• Before
  M4-T2: Orthogonal and Orthonormal Bases

  Find coordinates in an orthonormal basis with dot products alone, and explain why such bases are convenient.