Skip to the lesson
FixTheRobot Explore Robotics / Robot Learning / M2 Supervised Learning for Robots / M2-T3

Robot Learning · Theory · M2

M2-T3 — Loss Functions and Gradient Descent

A loss function says how wrong a prediction is; gradient descent repeatedly steps downhill on it. The step size has a hard limit, and the shape of the loss sets how fast you get there.

  • Lesson 3 of 6
  • Module 2: Supervised Learning for Robots
  • Theory
  • About 13 minutes to read

Lesson M2-T1 scored predictions; lesson M2-T2 chose the inputs. This lesson shows how a model is fitted: write a loss that measures error over the training data, compute its gradient with respect to the model’s weights, and move the weights a small step against it. For a linear model with squared error the loss is a quadratic bowl, which makes the whole story exact: a derived limit on the learning rate, a derived convergence rate in terms of the condition number, and numbers we can check by running the code. The same ideas carry, less cleanly, to the neural networks of lesson M2-T4. Lab M2-L1 has you write the gradient, the standardization and the training loop, and shows what the same learning rate does on raw and on standardized features.

Learning objectives

After this lesson, you should be able to:

  • Write the squared-error loss of a linear model and its gradient, and take a gradient descent step by hand.
  • Derive the stability condition lr < 2/λ_max for a quadratic loss and verify it numerically.
  • Explain how the condition number sets the convergence rate, and why standardising features helps.
  • Compare batch, stochastic and mini-batch gradients, and squared, absolute and Huber losses in the presence of outliers.

The intuition

Stand on a hillside in fog and want the valley floor. You feel which way the ground slopes under your feet and take a step downhill. If your steps are tiny, it takes forever. If they are too long on a narrow ridge, you cross the valley and land higher on the other side, then do it again, higher each time. That is gradient descent, and the overshoot is divergence.

For a quadratic bowl this has a clean description. Along each principal direction of the bowl the ground curves with some strength λ. One step multiplies the distance to the bottom, in that direction, by (1 − lr·λ). Any factor with magnitude below 1 shrinks; magnitude above 1 grows. The figure shows the three regimes in one dimension. The steepest direction (largest λ) sets the speed limit; the flattest (smallest λ) sets how long the trip takes. When they differ by a factor of 100, you must go slowly for the steep direction and crawl along the flat one. Standardising the features, lesson M2-T2, narrows that gap.

Gradient descent on a one-dimensional quadratic for three step sizesThe iterate starts at 1. With the product of learning rate and curvature at 0.5 it halves each step. At 1.9 it flips sign every step and shrinks by 0.9 each time. At 2.1 it flips sign and grows by 1.1 each step. lr·λ = 0.5: smooth decaylr·λ = 1.9: oscillates, still decayslr·λ = 2.1: oscillates and grows step tw1-1
The iterate on L(w) = ½λw² is multiplied by (1 − lr·λ) every step. Below lr·λ = 2 the magnitude shrinks, above it grows. Values are computed from that formula.

The mathematics

Squared-error loss and its gradient

For a linear model ŷ = x·w, training rows stacked into X (n × d) and targets y (n):

Squared error, gradient and Hessian
L(w)=(1n) ∑i(xi⋅w−yi)2=(1n) ∥X w−y∥2L(w)=(\frac{1}{n})\,\sum _{i}(x_{i}\cdot w-y_{i})^{2}=(\frac{1}{n})\,\lVert X\,w-y\rVert ^{2}
∇L(w)=(2n) X⊤(X w−y)\nabla L(w)=(\frac{2}{n})\,X^{\top}(X\,w-y)
H=∇2L=(2n) X⊤ XH=\nabla ^{2}L=(\frac{2}{n})\,X^{\top}\,X
constant: the loss is quadratic
w
weights (including the bias)
lr
learning rate (step size), written η in some books
∇L
gradient: the direction of steepest increase of the loss
H
Hessian: the matrix of second derivatives; its eigenvalues λ are the curvatures
w*
the minimiser, where ∇L = 0, i.e. Xᵀ X w = Xᵀ y (the normal equations)

Gradient descent repeats w ← w − lr·∇L(w). The gradient of a single row’s term is 2(xi·w − yi)·xi: the residual times the input. A row whose input is large pulls the weights harder, which is why feature scale matters.

The stability bound

Let et = wt − w*. Because ∇L(w) = H (w − w*), one step gives:

Derivation of the stability bound
et+1=et−lr⋅H et=(I−lr⋅H) ete_{t+1}=e_{t}-\mathrm{lr}\cdot H\,e_{t}=(I-\mathrm{lr}\cdot H)\,e_{t}
In the eigenbasis of H (eigenvalues λ1 ≥ … ≥ λd > 0), component i obeys
et+1, i=(1−lr⋅λi) et, ie_{t+1,\,i}=(1-\mathrm{lr}\cdot \lambda _{i})\,e_{t,\,i}
It shrinks for every i if and only if ∣1−lr⋅λi∣<1 for all i,\mathrm{It}\,\text{shrinks for every}\,i\,\text{if and only if}\,\lvert 1-\mathrm{lr}\cdot \lambda _{i}\rvert <1\,\text{for all}\,i,
i.e.0<lr<2λmax⁡.i.e.0<\mathrm{lr}<\frac{2}{\lambda _{\max}.}

This is exact for a quadratic loss. For a non-quadratic loss, such as a neural network, the Hessian varies and the bound is only local guidance (λmax can change along the path), but the habit of reading divergence as “lr too big for the steepest curvature” stays useful.

Convergence rate and the condition number

With lr = 1/λmax, the slowest component shrinks by 1 − 1/κ per step, where κ = λmax/λmin. The best fixed step, lr = 2/(λmax + λmin), balances the two extreme directions and gives the worst-case factor:

Best fixed-step convergence rate
rate=κ−1κ+1\text{rate}=\frac{\kappa -1}{\kappa +1}
per step, on the distance to w*

The excess loss L(w) − L(w*) shrinks like the square of that factor. At κ = 1 the rate is 0 (one step); at κ = 100 it is 0.98, so about 100 steps to gain a factor of a few.

Why standardisation helps

Standardising columns drives the diagonal of X′X toward equal values and, with the mean removed, the off-diagonals toward zero. The eigenvalues bunch together, κ drops, and one learning rate suits every direction (lesson M2-T2 computed κ = 101.9 raw and 1.20 standardised on Scout’s demonstration data).

Batch, stochastic and mini-batch

VariantGradient fromUpdates per epochBehaviour
Batchall n rows1exact gradient; smooth; costly for large n
Stochastic (SGD)1 rownnoisy; cheap per update; needs a small or decaying lr
Mini-batchB rows (e.g. 32)n / Bthe usual compromise; noise falls like 1/√B; vectorises well

Each stochastic gradient is an unbiased estimate of the full gradient, so on average it points the right way, but each one is noisy. With a constant learning rate the weights jitter around w* instead of settling on it; shrinking the step over time reduces the jitter. Shuffle the rows each epoch.

Other losses and outliers

squared
ℓ(r)=r2\ell (r)=r^{2}
gradient∝r\text{gradient}\propto r
(unbounded)(\text{unbounded})
absolute
ℓ(r)=∣r∣\ell (r)=\lvert r\rvert
gradient=sign⁡(r)(bounded; fit follows the median)\text{gradient}=\operatorname{sign} (r)(\text{bounded};\,\text{fit follows the median})
Huber(δ)
ℓ(r)=12r2\ell (r)=\tfrac{1}{2}r^{2}
if ∣r∣≤δ\text{if}\,\lvert r\rvert \le \delta
ℓ(r)=δ(∣r∣−12δ)\ell (r)=\delta (\lvert r\rvert -\tfrac{1}{2}\delta )
if ∣r∣>δ\text{if}\,\lvert r\rvert >\delta
gradient=clip⁡(r, −δ, δ)\text{gradient}=\operatorname{clip} (r,\,-\delta ,\,\delta )

Squared error lets one bad row pull the fit with a force proportional to its error. Absolute loss limits that pull to a constant, but its gradient does not shrink near the minimum and it is not smooth at 0. Huber loss is quadratic for small errors and linear for large ones, giving the smooth, fast behaviour of squared error on typical rows and the robustness of absolute loss on wild ones. The threshold δ is in the units of y and is a choice.

Loss is not metric

The loss is what you can optimise; the metric (lesson M2-T1, or the success rate of module M1) is what you care about. They differ, and a lower loss need not give a higher success rate.

A worked example

Numbers below were computed by running code. Part 1 is small enough to check by hand; parts 2 to 4 use the 945 Scout demonstration pairs of lesson M2-T2.

1. Three gradient steps by hand

Fit ŷ = a + b·x to the six steering pairs of lesson M2-T1: x = (−0.6, −0.3, 0, 0.2, 0.5, 0.8), y = (0.45, 0.35, 0.05, −0.15, −0.30, −0.65). Loss L = mean of (a + b x − y)². Start at (a, b) = (0, 0) with lr = 0.5.

  1. At (0, 0) the residuals are −y. ∂L/∂a = 2·mean(r) = 2·(+0.0417) = 0.0833. ∂L/∂b = 2·mean(r·x) = 0.3583. L = 0.1438. Update: a = 0 − 0.5·0.0833 = −0.0417, b = 0 − 0.5·0.3583 = −0.1792. New L = 0.0867.
  2. Gradient at (−0.0417, −0.1792): (−0.0358, 0.2676). Update: a = −0.0237, b = −0.3130. L = 0.0542.
  3. Gradient at (−0.0237, −0.3130): (−0.0268, 0.2096). Update: a = −0.0104, b = −0.4178. L = 0.0343.

The loss falls 0.1438 → 0.0867 → 0.0542 → 0.0343, heading to the least-squares optimum a = 0.0379, b = −0.7955 with L* = 0.0028. After three steps b has covered about half of its distance, while a is already close. The Hessian is H = [[2, 0.2], [0.2, 0.46]] with eigenvalues λmax = 2.0256 and λmin = 0.4344, so κ = 4.66. With lr = 0.5 the per-step factors are 1 − 0.5·2.0256 = −0.013 (steep direction: gone in one step) and 1 − 0.5·0.4344 = 0.783 (flat direction: slow). That is why b lags.

2. Verifying the limit 2/λmax

The bound for this problem is 2/2.0256 = 0.987. Running 30 steps from (0, 0):

lrFraction of boundLoss after 30 steps
0.8890.90.002809 (the optimum, 0.0028093)
0.9871.00.00686 (stuck oscillating)
1.0861.1228.1 (diverging)

The same test on the real data, target ax from features [dx, dy, 1] with H = (2/945) X′X, 200 steps each:

FeaturesλmaxκBound 2/λmax0.95 × bound1.05 × bound
Raw (metres)41.35101.90.0484excess loss 7.8 × 10⁻⁵excess loss 2.3 × 10¹⁶
Z-scored2.1801.1970.918excess loss 1 × 10⁻¹⁷excess loss 8.2 × 10¹²

The predicted threshold is sharp: 5% below, converged to within rounding or nearly; 5% above, blown up. Setting lr = 0.05 on the raw features crosses the line 0.0484 and blows up, which is the first thing to check when a loss curve rises. Lab M2-L1 shows a second cause: a loop that reuses a stale gradient also makes the loss dive and then rise.

3. The condition number sets the step count

Stop when 99.9% of the initial excess loss is gone. At lr = 1/λmax the raw features take 265 steps (rate factor 1 − 1/κ = 0.990 in the slowest direction); the z-scored features, with lr = 0.459, take 2. The best-step rate formula (κ − 1)/(κ + 1) gives 0.981 for the raw set and 0.090 for the z-scored set. Standardising did not change the best fit (the minimum loss is 0.010824 for both); it changed how fast and how safely we reach it.

4. Batch, stochastic and mini-batch: one epoch on z-scored data

Each variant sees all 945 rows once (945 per-row gradient evaluations). Learning rates were set by hand, not tuned, so this compares behaviour, not a winner. Row order shuffled with seeds 1, 2, 3. The optimum loss is 0.010824.

VariantlrUpdatesLoss after one epoch (seeds 1, 2, 3)
Batch (945)0.310.1361 (identical for all seeds)
Mini-batch (32)0.1300.01086, 0.01112, 0.01100
SGD (1)0.019450.01087, 0.01126, 0.01185

Same data cost, very different progress: one big step against 30 or 945 small ones. The stochastic variants have already reached within about 0.4% to 10% of the optimum, and they differ by seed, a visible sign of gradient noise. The batch variant needs more epochs.

5. Outliers: squared against Huber

Add one corrupted row to the six pairs: (x, y) = (0.3, 0.9), when the line says about −0.2 (a residual of 1.10 against the clean fit; a sign error in a logged command). Squared error moves the fit to a = 0.177, b = −0.656 and predicts −0.151 at x = 0.5, against −0.360 before. Huber with δ = 0.1, run by gradient descent until it stopped moving, gives a = 0.053, b = −0.780 and predicts −0.337 at x = 0.5. One row out of seven shifted the squared-error prediction by 0.21 and the Huber prediction by 0.02. On the clean six rows Huber with δ = 0.1 returns exactly the least-squares line, because every residual is below δ.

What this simplifies. The loss is exactly quadratic, so the bounds are exact; for a neural network or a non-convex loss they hold only locally. The batch, mini-batch and SGD numbers come from three shuffles and hand-set learning rates and should be read as illustrations, not as a benchmark. The corrupted row is a single hand-made outlier, not a model of real sensor faults.

Engineering implications and limitations

Debug divergence in a fixed order

Loss goes to infinity or NaN: (1) print the loss at the first ten steps; if it grows, the learning rate is too high or features are unscaled. (2) Standardise the inputs. (3) Reduce the learning rate by 3 or 10. (4) Check the gradient against a finite-difference estimate on a few weights; a sign or factor-2 bug looks exactly like divergence.

Check the gradient numerically

Compare ∂L/∂wj with (L(w + h ej) − L(w − h ej))/2h at h around 1e-5. If they agree to 5 or 6 digits, the code is right. Do this once for every new loss.

Estimate λmax when you can

For a linear model it is the top eigenvalue of (2/n) X′X; power iteration finds it in a few lines. Pick lr at about half the bound, or at 1/λmax.

Plot the loss on a log scale

A straight line on a log plot is geometric convergence at a fixed rate; the slope is your effective rate. A zigzag that grows is the stability bound crossed.

Choose the batch size on cost and noise

Larger batches give smoother gradients but fewer updates per epoch. With a fixed compute budget, 32 to 256 is a common range; keep the learning rate tied to the batch size and re-tune when you change one.

Choose the loss to match the data

If logged commands contain occasional wild values (operator mis-clicks, sensor dropouts), try Huber or clip the targets first, and look at the rows with the largest residuals before deciding whether they are errors or rare real events. Dropping rare real events (an emergency stop) deletes the behaviour you most need.

Common misconceptions

“A smaller learning rate is always safe.”

Safe from divergence, but costly: on the raw features at lr = 1/λmax the fit needed 265 steps and a rate 100 times smaller would need tens of thousands. It can also stop the search early, mistaken for convergence.

“If the loss goes down, the policy gets better.”

The loss measures fit to logged actions. Lesson M1-T6 shows 0.0593 MSE giving 45% success while 0.0088 gave 100%. Check the metric you care about as well.

“Divergence means the data are bad.”

For a quadratic loss it means exactly lr > 2/λmax. Rescale the features or reduce the step before suspecting the data.

“SGD is just a worse gradient descent.”

Each update is cheaper and noisy but unbiased; with 945 rows one epoch of SGD did 945 updates and reached within about 10% of the optimum. The noise also needs a smaller or decaying step.

Engineering takeaways

  1. Squared error L = (1/n)‖Xw − y‖² has gradient (2/n)Xᵀ(Xw − y) and constant Hessian (2/n)XᵀX.
  2. Gradient descent on a quadratic is stable if and only if lr < 2/λ_max, checked on Scout data: 0.0484 raw, 0.918 standardised.
  3. The best convergence factor is (κ − 1)/(κ + 1); standardising cut κ from 101.9 to 1.2 and the steps from 265 to 2.
  4. Mini-batches trade gradient noise for more updates per epoch; Huber loss limits the pull of outliers (one bad row moved the squared-error prediction by 0.21, Huber by 0.02).
  5. The loss is a proxy for the metric you care about; read both, and see lab M2-L1 for loops that go wrong in both ways.

References and further reading

Standard sources on optimisation, practical training advice and robust losses.

  • S. Boyd and L. Vandenberghe, “Convex Optimization”, Cambridge University Press, 2004.
  • Y. LeCun, L. Bottou, G. B. Orr and K.-R. Müller, “Efficient BackProp”, in Neural Networks: Tricks of the Trade, Springer, 1998.
  • P. J. Huber, “Robust Estimation of a Location Parameter”, Annals of Mathematical Statistics, 35(1), 1964.
  • L. Bottou, F. E. Curtis and J. Nocedal, “Optimization Methods for Large-Scale Machine Learning”, SIAM Review, 60(2), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A quadratic loss has Hessian eigenvalues 4 and 0.1. What is the largest stable learning rate and the condition number?

Explanation

lr < 2 / λ_max = 2 / 4 = 0.5, and κ = 4 / 0.1 = 40.

Question 2

You standardise the features and the same learning rate now diverges less, or even allows a 20-times larger value. Why?

Explanation

Standardising lowers the condition number and removes huge curvatures from large-valued features, so λmax can drop and 2/λmax rises. One learning rate then suits all directions. On Scout data the bound moved from 0.0484 to 0.918.

Question 3

One row in a regression log has a wild target. Name two ways to limit its influence without deleting it.

Explanation

Use a robust loss such as Huber (gradient clipped at δ) or absolute loss, or clip the target to the plausible range. Check first whether the row is an error or a rare real event.

What’s next

Next: lesson M2-T4 builds a small neural network from scratch. Its gradients come from the chain rule applied to the same squared-error loss, and the learning-rate and conditioning issues of this lesson return with it.

Practice this lesson

Fit Steering with Gradient Descent

Predict Scout's steering command from its state with a linear model trained by gradient descent that currently diverges.

Next lesson

M2-T4 — Neural Networks from Scratch

Write the forward pass and the gradients of a small multilayer network, and say what each layer adds.

Nothing below is required to finish this lesson.