M4-T6 — Quadratic Forms

Learning objectives

After this lesson, you should be able to:

• Write a quadratic form xᵀAx, expand it for 2 × 2 and recognise why A can be taken symmetric.
• Classify a symmetric 2 × 2 matrix as positive definite, negative definite, semidefinite or indefinite with the a > 0 and ac − b² > 0 test.
• Picture the level curves as ellipses (positive definite) or hyperbolas (indefinite).
• Relate quadratic forms to least-squares cost, kinetic energy and covariance.

Introduction and motivation

The squared error of a fit, the kinetic energy of an arm, the cost an optimiser minimises and the Mahalanobis distance used to gate a measurement are all of the form xᵀAx for some symmetric matrix A. Whether that expression is always positive, and what its level curves look like, decides whether an optimiser has a unique answer, whether a covariance is legitimate and whether a robot's energy makes sense. It is worth meeting the idea in the simplest setting before Module 5, Module 6 and Module 9 use it.

The intuition

A quadratic form takes a vector and returns a single number built from squares and products of its entries: q(x, y) = a x² + 2b xy + c y². Plot it as a height above the plane. If it is always positive (except at the origin) the graph is a bowl. If it dips below zero in some directions and rises above in others it is a saddle. If it is always negative it is a dome.

Geometric and physical interpretation

Geometric view. The set where xᵀAx = 1 is an ellipse when A is positive definite, with axes along the eigenvectors of A and semi-axis lengths 1/√λ: a large eigenvalue means the form grows quickly in that direction, so the level curve is close to the origin. It is a pair of hyperbolas when A is indefinite. Physical view. Moving the arm in a direction that costs a lot of kinetic energy per unit speed is a direction with a large eigenvalue of the mass matrix; the ellipse is the set of equal-energy velocities.

The mathematics

q(x)

the quadratic form x ↦ xᵀAx, a single number.

A

a symmetric matrix (Aᵀ = A); in 2D [[a, b], [b, c]].

λ

an eigenvalue of A (Module 6); the signs of the eigenvalues decide the type.

The form

q(x, y) = [x y] [a b; b c] [x; y] = a x² + 2 b x y + c y²

What it means. Each diagonal entry multiplies a squared coordinate; the off-diagonal entry multiplies the cross term xy. The factor 2 appears because b occurs twice in the product, above and below the diagonal.

Only the symmetric part matters

xᵀ A x = xᵀ S x     with   S = (A + Aᵀ)/2

What it means. The skew-symmetric part contributes nothing to the form, so we always take A symmetric when talking about quadratic forms.

Definiteness

positive definite:   xᵀAx > 0 for all x ≠ 0        (2 × 2: a > 0 and a c − b² > 0)
negative definite:   xᵀAx < 0 for all x ≠ 0
indefinite:          xᵀAx takes both signs        (2 × 2: a c − b² < 0)
semidefinite:        xᵀAx ≥ 0 (or ≤ 0), with zero for some x ≠ 0

What it means. Positive definite means a bowl with a single minimum at 0. For 2 × 2 matrices two inequalities decide it, derived below by completing the square. The same classification is read from the signs of the eigenvalues.

xᵀAx is a bowl, saddle or dome. Positive definite means always positive: a unique minimum and elliptical level curves.

Step-by-step derivation: the symmetric part, and the 2 × 2 test by completing the square

Only the symmetric part matters. Split A = S + K with S symmetric and K skew-symmetric (Module 2). Then xᵀKx is a single number, so it equals its own transpose: xᵀKx = (xᵀKx)ᵀ = xᵀKᵀx = −xᵀKx. A number equal to its negative is zero, so xᵀKx = 0 and xᵀAx = xᵀSx.

The 2 × 2 expansion. With A = [[a, b], [b, c]], xᵀAx = a x² + 2b xy + c y². Now complete the square, assuming a ≠ 0:

1. Factor a out of the first two terms: a x² + 2b xy = a (x² + (2b/a) xy) = a ((x + (b/a) y)² − (b²/a²) y²).
2. So q = a (x + (b/a) y)² − (b²/a) y² + c y² = a (x + (b/a) y)² + (c − b²/a) y².
3. This is a sum of two squares with coefficients a and c − b²/a = (ac − b²)/a. The form is positive for all non-zero (x, y) exactly when both coefficients are positive.
4. a > 0 and (ac − b²)/a > 0, which for a > 0 is ac − b² > 0. That is the test: positive definite ⇔ a > 0 and ac − b² > 0.
5. If ac − b² < 0 the two coefficients have opposite signs (whatever the sign of a), so q takes both signs: indefinite.

Why the level curve is an ellipse. After the change of variables u = x + (b/a)y, w = y, the equation q = 1 reads a u² + ((ac − b²)/a) w² = 1, an ellipse in (u, w) when both coefficients are positive. Module 6 shows that rotating to the eigenvector axes makes this even cleaner: λ₁ s² + λ₂ t² = 1 with semi-axes 1/√λ₁ and 1/√λ₂.

A worked example: classifying two forms

A positive definite form. A = [[2, 1], [1, 3]], so q(x, y) = 2x² + 2xy + 3y².

1. Test. a = 2 > 0 and ac − b² = 6 − 1 = 5 > 0: positive definite.
2. Complete the square. q = 2(x + y/2)² + (3 − 1/2) y² = 2(x + y/2)² + 2.5 y², a sum of two positive terms.
3. Evaluate. q(1, −1) = 2 − 2 + 3 = 3 and q(1, 0) = 2, both positive as promised.
4. Eigenvalues. λ = (5 ± √5)/2 ≈ 3.618 and 1.382, both positive. The level curve q = 1 has semi-axes 1/√3.618 = 0.526 and 1/√1.382 = 0.851.

An indefinite form. A = [[1, 2], [2, 1]]: ac − b² = 1 − 4 = −3 < 0. Indeed q(1, 1) = 1 + 4 + 1 = 6 > 0 while q(1, −1) = 1 − 4 + 1 = −2 < 0. The level curves are hyperbolas, and the origin is a saddle.

The symmetric part. For the non-symmetric M = [[2, 3], [−1, 3]] the form is xᵀMx = 2x² + 2xy + 3y², identical to the first example, because S = (M + Mᵀ)/2 = [[2, 1], [1, 3]].

Robotics application

Least-squares cost. The squared error of a fit, ‖Ax − b‖² = xᵀ(AᵀA)x − 2bᵀAx + bᵀb, is a quadratic function of x whose matrix AᵀA is positive definite when the columns of A are independent. A bowl has exactly one bottom, which is why a well-posed fit has one best answer (Module 5).

Energy. The kinetic energy of a manipulator is ½ q̇ᵀ M(q) q̇ with a positive definite mass matrix M; the ellipse q̇ᵀMq̇ = const shows which joint velocity directions are cheap and which are expensive. (This is also why a physical mass matrix can never have a non-positive eigenvalue.)

Uncertainty. The squared Mahalanobis distance xᵀΣ⁻¹x of Module 9 is a quadratic form whose level curves are the uncertainty ellipses drawn around a robot's estimated position; gating a measurement means testing which ellipse it falls in. See probability and uncertainty for the estimation context.

Optimisation. Near a minimum a smooth cost looks like a quadratic form (the Hessian); a positive definite Hessian means a true minimum, an indefinite one a saddle, where gradient methods can stall. The nonlinear least squares lesson solves such problems iteratively.

Engineering implications and limitations

Practical rules

• Test positive definiteness with a Cholesky attempt. If the Cholesky factorisation of a symmetric matrix succeeds, it is positive definite; if it fails, it is not. It is cheap and robust. (Checking determinants of larger matrices is unreliable numerically.)
• Symmetrise first. A covariance or Hessian that is symmetric only up to rounding should be replaced by (A + Aᵀ)/2 before any definiteness test.
• A tiny negative eigenvalue is usually rounding. A covariance that should be positive semidefinite can show eigenvalues of −10⁻¹⁶. Clip them to zero, or add a small multiple of the identity, rather than letting the filter fail.
• Scale matters for the ellipse picture. The shape of the level curve depends on the units; a form mixing metres and radians has an ellipse whose axes are not comparable.
• Semidefinite means a flat direction. Zero eigenvalues mean the form does not penalise some direction at all: the optimisation then has infinitely many answers or the covariance has a certain direction (a degeneracy).

Limits

The definiteness classification uses the ordinary squared length. A quadratic form can be positive in one coordinate system and have a different shape in another (it is the same form, but the matrix changes), so judge definiteness from invariants (eigenvalues), not from entries.

Common misconceptions

Engineering takeaways

1. A quadratic form is q(x) = xᵀAx = a x² + 2b xy + c y² in 2D, and only the symmetric part of A matters.
2. Positive definite means q > 0 for all x ≠ 0; for 2 × 2 symmetric matrices this is a > 0 and ac − b² > 0.
3. The level curves are ellipses for a positive definite form and hyperbolas for an indefinite one; the axes follow the eigenvectors.
4. Least-squares cost, kinetic energy and Mahalanobis distance are quadratic forms.
5. Test definiteness with a Cholesky attempt or the eigenvalues, after symmetrising.

References and further reading

Primary papers and authoritative documentation.

• MIT 18.06SC Linear Algebra (OCW): lectures on positive definite matrices and quadratic forms
• Mathematics for Machine Learning: the chapter on matrix decompositions and on optimisation: positive definite matrices
• Modern Robotics (Lynch and Park), free preprint: chapter 8 appendix: the mass matrix as a quadratic form for kinetic energy
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS): chapters on least squares: the cost is a quadratic function; the site did not load when we last checked

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Module 4 gave you the geometry of right angles: orthogonal sets, projections, orthonormal frames and quadratic forms. Module 5 uses all of it to answer the question robots face every day: what is the best answer when no exact answer exists?

Related content

Nothing below is required to finish this lesson.

• Before
  M4-T5: Orthogonal Matrices

  State the properties of an orthogonal matrix, including that it preserves lengths and angles, and relate them to rotations and reflections.