M6-T5 — Singular Value Decomposition

Learning objectives

After this lesson, you should be able to:

• State the SVD A = UΣVᵀ for any m × n matrix and name the shapes and properties of U, Σ and V.
• Derive the SVD from the eigen-decomposition of AᵀA.
• Interpret the singular values geometrically as the semi-axes of the image of the unit sphere.
• Read the rank, the norm and the condition number from Σ, and relate the SVD to the four fundamental subspaces.

Introduction and motivation

A robot arm Jacobian is rectangular. A point-cloud cross-covariance matrix is not symmetric. A camera matrix has no eigenvectors worth looking at. Yet all of them can be understood with the same picture: an ellipse. The SVD computes that ellipse for any matrix and is the numerically safest way to answer the questions of Modules 3, 5 and 8: what is the rank, is the problem well conditioned, which directions are weak, and what is the best low-rank or least-squares answer.

The intuition

Take the unit circle of inputs and push it through the matrix. You always get an ellipse (possibly flattened to a line). The longest axis of that ellipse is the biggest stretch the matrix can produce; the input direction that produces it is v₁, the output direction is u₁ and the stretch is σ₁. The next axis, perpendicular to the first, is the second biggest stretch, and so on. Those axes are the SVD.

Geometric and physical interpretation

Geometric view. The columns of V are an orthonormal basis of input space chosen so that A maps it to perpendicular outputs; the columns of U are those outputs, normalised; the singular values are the lengths. For a wide or tall matrix, some right singular vectors map to zero (the null space) and some left singular vectors are never reached (the left null space). Physical view. For a Jacobian, the SVD lists the directions in which the hand can move easily (large σ) and with difficulty (small σ); the ellipse is the manipulability ellipse of Module 8.

The mathematics

A

an m × n real matrix, any shape.

U

m × m orthogonal; columns uᵢ are the left singular vectors.

V

n × n orthogonal; columns vᵢ are the right singular vectors.

Σ

m × n diagonal (rectangular) with σ₁ ≥ σ₂ ≥ … ≥ 0 on the diagonal: the singular values.

r

the rank: the number of non-zero singular values.

The decomposition

A = U Σ Vᵀ = σ₁ u₁v₁ᵀ + σ₂ u₂v₂ᵀ + ... + σᵣ uᵣvᵣᵀ

What it means. Every matrix is a sum of r rank-one pieces, ordered by importance. A vᵢ = σᵢ uᵢ: A sends the input axis vᵢ to the output axis uᵢ, scaled by σᵢ.

Connection to eigenvalues

AᵀA = V Σᵀ Σ Vᵀ,     A Aᵀ = U Σ Σᵀ Uᵀ,     σᵢ² = eigenvalues of AᵀA

What it means. The right singular vectors are the eigenvectors of AᵀA and the left ones those of AAᵀ; the squares of the singular values are their (shared, non-negative) eigenvalues. This is both a derivation and a (numerically inferior) way to compute the SVD.

Norm, rank and conditioning

‖A‖₂ = σ₁      rank(A) = #{σᵢ > 0}      cond(A) = σ₁ / σᵣ

What it means. The largest singular value is the biggest stretch (the 2-norm). The rank counts the non-zero ones. The ratio of the largest to smallest non-zero one is the condition number, the sensitivity of solving Ax = b.

Every matrix = rotate, stretch by σ₁ ≥ σ₂ ≥ … ≥ 0, rotate. The σᵢ² are the eigenvalues of AᵀA; the number of non-zero σᵢ is the rank.

Step-by-step derivation: building the SVD from AᵀA

1. AᵀA is symmetric and positive semi-definite (xᵀAᵀAx = ‖Ax‖² ≥ 0), so by the spectral theorem AᵀA = V Λ Vᵀ with orthonormal eigenvectors vᵢ and eigenvalues λᵢ ≥ 0.
2. Define the singular values σᵢ = √λᵢ, sorted from large to small. Then ‖A vᵢ‖² = vᵢᵀAᵀA vᵢ = λᵢ = σᵢ², so A vᵢ has length σᵢ.
3. For every σᵢ > 0, define uᵢ = A vᵢ / σᵢ, a unit vector.
4. The uᵢ are orthogonal: uᵢᵀuⱼ = vᵢᵀAᵀA vⱼ/(σᵢσⱼ) = λⱼ vᵢᵀvⱼ/(σᵢσⱼ) = 0 for i ≠ j.
5. Collecting A vᵢ = σᵢ uᵢ for all i gives A V = U Σ (complete U with extra orthonormal vectors for the zero singular values). Since V⁻¹ = Vᵀ, A = U Σ Vᵀ.

Consequences. If σᵢ = 0 then ‖A vᵢ‖ = 0, so vᵢ is in the null space; the right singular vectors with σᵢ = 0 span the null space and those with σᵢ > 0 span the row space. The left singular vectors with σᵢ > 0 span the column space and the remaining ones the left null space. The SVD is therefore an orthonormal basis for all four fundamental subspaces of Module 3 at once.

Best low-rank approximation. Keeping the first k terms, Aₖ = Σᵢ≤ₖ σᵢ uᵢvᵢᵀ, gives the closest rank-k matrix to A in the 2-norm and the Frobenius norm (Eckart-Young theorem), with error σₖ₊₁ in the 2-norm. This is the principle behind PCA and data compression.

Uniqueness. The singular values are unique. Each pair (uᵢ, vᵢ) is unique up to a joint sign flip, unless singular values repeat, in which case any orthonormal basis of the corresponding subspace may be used.

A worked example: the SVD of a 2 × 2 matrix

Let A = [[3, 0], [4, 5]].

• Form AᵀA. AᵀA = [[9 + 16, 0 + 20], [20, 25]] = [[25, 20], [20, 25]].
• Eigen-decompose it. Trace 50, determinant 625 − 400 = 225, so λ² − 50λ + 225 = 0, giving λ = 45 and 5. Eigenvectors: (1, 1)/√2 for 45 and (1, −1)/√2 for 5.
• Singular values. σ₁ = √45 = 6.708, σ₂ = √5 = 2.236. Check: their product 6.708 × 2.236 = 15 = |det A| ✓, and σ₁² + σ₂² = 50 = ‖A‖²F = 9 + 0 + 16 + 25 ✓.
• Left vectors. u₁ = A v₁/σ₁ = (3, 9)/√2 / 6.708 = (0.3162, 0.9487); u₂ = A v₂/σ₂ = (3, −1)/√2 / 2.236 = (0.9487, −0.3162). They are perpendicular ✓ (up to the sign convention a library picks).
• Condition number. σ₁/σ₂ = 6.708/2.236 = 3.0: the matrix stretches three times more in one direction than the other, a well-conditioned map.
• Rebuild. σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ returns A; the rank-one truncation σ₁u₁v₁ᵀ = [[1.5, 1.5], [4.5, 4.5]] is the best rank-one approximation, with error σ₂ = 2.236.

Robotics application

Point-cloud registration (ICP). The optimal rotation aligning two clouds comes from the SVD of the 3 × 3 cross-covariance matrix H = Σ pᵢ qᵢᵀ: R = V diag(1, 1, det(VUᵀ)) Uᵀ (Kabsch/Umeyama). See ICP fundamentals and scan registration.

Manipulators. The singular values of the Jacobian give the speed the hand can reach in each direction; a vanishing σ marks a singularity. Module 8 uses this directly.

Plane and normal estimation from LiDAR. The right singular vector for the smallest singular value of the centred point matrix is the surface normal (next lessons on rank and PCA).

Stable least squares. Solving Ax ≈ b through the SVD never squares the condition number and handles rank deficiency, unlike the normal equations. UAV sensor calibration and camera calibration (homography, essential matrix) all use the SVD to obtain null vectors; see extrinsic calibration.

Engineering implications and limitations

Practical rules

• Use numpy.linalg.svd, never AᵀA, for accuracy. Squaring A squares the condition number and destroys small singular values. Compute the SVD of A itself.
• Choose full_matrices=False for tall matrices. The economy SVD returns only the first n columns of U; it is far cheaper and enough for least squares and PCA.
• Singular values are non-negative and sorted. Signs live in U and V; never expect a “negative singular value”.
• Pair signs consistently. If you flip uᵢ you must flip vᵢ. Do not compare vectors between runs without fixing a sign convention.
• Cost. A dense SVD is O(mn²) for m ≥ n; for a 3 × 3 or 6 × 6 matrix inside a real-time loop this is trivial, for big matrices consider truncated or randomised SVD.
• Determinant sign. A rotation recovered by U Vᵀ may have determinant −1 (a reflection); correct it as in the registration formula above.

Limits

The SVD describes a linear map. For a nonlinear system it describes only the local Jacobian, valid in a small neighbourhood. It also does not respect physical weighting: if components have different units, scale them first (Module 5, weighted least squares), otherwise the singular values compare metres with radians.

Common misconceptions

Engineering takeaways

1. Every matrix factors as A = UΣVᵀ: rotate, stretch along perpendicular axes by σ₁ ≥ σ₂ ≥ … ≥ 0, rotate.
2. The σᵢ² are the eigenvalues of AᵀA and AAᵀ; A vᵢ = σᵢ uᵢ.
3. σ₁ = ‖A‖₂, rank = number of non-zero σ, condition number = σ₁/σᵣ.
4. The SVD gives orthonormal bases of all four fundamental subspaces and the best rank-k approximation.
5. It underlies ICP alignment, manipulability, PCA, pseudoinverses and stable least squares; compute it from A, not AᵀA.

References and further reading

Primary papers and authoritative documentation.

• MIT 18.06SC Linear Algebra (OCW): lecture on the singular value decomposition
• MIT 18.06SC Linear Algebra (OCW): lecture on image compression and the best rank-k approximation
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): the SVD and its uses, where covered; the site did not load when we last checked
• Mathematics for Machine Learning: chapter 4, matrix decompositions: the SVD
• UCSD ECE276A: Sensing and Estimation in Robotics: SVD in rigid registration and the Kabsch algorithm

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The singular values contain the answer to two practical questions at once: how many independent directions does the data really have, and how close to singular is the problem. The next lesson turns them into the numerical rank and the condition number.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T4: Positive Definite Matrices

  Test whether a symmetric matrix is positive definite and explain why covariance matrices must be.

• Before
  M4-T5: Orthogonal Matrices

  State the properties of an orthogonal matrix, including that it preserves lengths and angles, and relate them to rotations and reflections.

• Lab
  M6-L1: Point Cloud PCA

  Find the surface normal and principal axes of a point cloud with the SVD, and tell planes, edges and volumes apart.