M2-T4 — Neural Networks from Scratch

Learning objectives

After this lesson, you should be able to:

• Write the forward pass of a two-layer tanh network in matrix form and count its parameters.
• Derive the gradients of the mean squared error for every weight and bias with the chain rule.
• Check an implementation of backpropagation against finite differences and read the relative error.
• Explain what hidden units add that a linear model lacks, and why a network is still a fitted curve and not magic.

The intuition

A linear model draws one straight line through its inputs. If the right answer bends, no choice of the line’s two numbers will follow it. The fix is to bend the model: compute several straight-line responses, pass each through a squashing function such as tanh, and add them up with weights. Each squashed response is a smooth step whose position, steepness and height are parameters. A handful of steps, added, can trace an S-shaped curve.

That is a neural network with one hidden layer. Training means moving every position, steepness and height downhill on the loss. The only new difficulty is bookkeeping: a change in an early weight reaches the loss through every later layer, so its gradient is a product of local slopes. Backpropagation is that product, computed from the loss backwards so each local slope is used once.

The mathematics

The neuron and the layer

A neuron computes a weighted sum of its inputs plus a bias, then applies a fixed nonlinearity φ. A layer is several neurons that share the same inputs, which is a matrix multiplication:

Two nonlinearities cover most small robot models. tanh(z) squashes to (−1, 1), is smooth, and has slope 1 − tanh²(z); it saturates (slope near 0) when |z| is large. ReLU(z) = max(0, z) has slope 1 for z > 0 and 0 otherwise; it does not saturate on the positive side, but a unit whose input is negative for every sample has zero gradient and stops learning (a “dead” unit).

The forward pass of a two-layer network

For one input x with d inputs, H hidden units and m outputs, with a linear output layer for regression:

For a batch of B samples stack the inputs as columns of a d × B matrix; the same equations hold with the biases added to every column, and the loss is the average over the batch. The ½ makes the derivative clean; some libraries use the plain mean of squares, which doubles every gradient below (lesson M2-T3).

Backpropagation, every gradient

Work backwards from the loss. Each line uses the line before it:

For a batch, sum each product over the samples and divide by B. For ReLU replace (1 − h²) by the indicator [z1 > 0]. Gradient descent then updates every parameter by θ ← θ − η ∂L/∂θ with a step size η (lesson M2-T3).

δ

the “error signal” at a layer: the gradient of the loss with respect to that layer’s pre-activation z.

⊙

elementwise product.

d, H, m

input size, number of hidden units, output size.

η

the step size (learning rate).

How many parameters?

Count them before training: the number of parameters against the number of independent samples is the first guide to overfitting (lesson M2-T5).

Initialisation

Weights must start different from each other and not too large. If every hidden unit starts identical, they receive identical gradients and stay identical forever; if they start large, tanh saturates and its slope 1 − h² is near zero, so nothing learns. Two standard rules scale the random draw by the layer size: Glorot uniform draws from ±sqrt(6 / (fan_in + fan_out)), which is ±0.707 for a 4 → 8 layer; He normal uses standard deviation sqrt(2 / fan_in) and suits ReLU. Biases usually start at zero. Normalise the inputs first (lesson M2-T2) or the first layer saturates on its own.

Gradient checking

A backpropagation bug rarely crashes; it just trains badly. Test it by comparing each analytic gradient with a central finite difference of the loss:

In double precision a correct implementation gives a relative error below about 1e−6. Values near 1e−2 or worse mean a bug. Do the check once on a tiny network and a few samples, then turn it off, because it costs two forward passes per parameter.

A worked example

All numbers were computed by running the formulas above in Node (double precision) and are rounded to four decimals for display.

1. One forward and one backward pass. A 2 → 2 → 1 tanh network with input x = (0.5, −1) and target y = 0.9. Parameters: W1 = [[0.4, −0.2], [0.1, 0.3]], b1 = (0.1, −0.1), W2 = [0.5, −0.4], b2 = 0.2.

2. The gradient check. Perturb each of the nine parameters by ε = 1e−5 and compare. For W1[0][1] the analytic gradient is 0.131491 and the central difference is 0.131491; for W2[0] they are −0.154528 and −0.154528; for b2, −0.334391 and −0.334391. Over all nine parameters the largest relative error is 2.3e−11, far below the 1e−6 line, so the formulas are right.

3. What a bug looks like. Drop the tanh slope (use ∂L/∂h in place of δ1, a common slip). The W1[0][1] gradient becomes 0.1672 instead of 0.1315, a relative error of 0.12, a hundred thousand times above the tolerance. The network would still train, only more slowly and to a worse place; the check is the only thing that catches it.

4. One update. With η = 0.5, W2 becomes (0.5773, −0.4562), b2 = 0.3672, W1 = [[0.4329, −0.2657], [0.0703, 0.3593]] and b1 = (0.1657, −0.1593). The prediction moves from 0.5656 to 0.9012 and the loss from 0.0559 to 7.8e−7. One sample can be fitted almost exactly in one step; with many samples the steps for different samples pull against each other and the loss falls more slowly.

5. What capacity adds. Take 40 evenly spaced points of the curved corridor y = 0.8 sin(2x) on [−2, 2], noise-free. The least-squares line has slope 0.107 and a mean squared error of 0.267, against a variance of the targets of 0.283: it explains 6% of the variation. Tanh networks with H hidden units (1 → H → 1, 3H + 1 parameters) trained by 20,000 full-batch gradient-descent steps with η = 0.3, median of five seeds:

Four units follow the curve. Eight units fit worse in the same 20,000 steps, not because they are less capable but because this budget and step size did not finish the optimisation: the best of five seeds for H = 8 reached 1.9e−3 while a four-unit seed reached 1.6e−5. More capacity is a possibility, not a guarantee; training must still find it.

What the simulation simplifies. These are noise-free one-dimensional curves with full-batch gradient descent. Real robot data are noisy, higher-dimensional and trained with minibatches and adaptive step sizes; the derivation is the same, the behaviour is messier. Training error says nothing yet about new data; that is lesson M2-T5.

Engineering implications and limitations

Build it, check it, then use a library

Write the forward and backward pass by hand once (lab M2-L2 does this), check it against finite differences, then switch to an automatic-differentiation library for anything larger. The library computes the same gradients; you will know what to suspect when a loss does not fall.

Normalise inputs and targets

A first layer fed raw positions in metres and speeds in millimetres per second saturates at once. Standardise inputs with statistics from the training set only (lesson M2-T6), and scale the targets to a similar range.

Watch the training curve

A loss that does not fall at all means a step size far too small, saturated units or a gradient bug. A loss that jumps to infinity means the step is too large. Plot the loss every few hundred steps; a table of final numbers hides both.

Start small

Begin with the smallest network that can plausibly fit, a few dozen hidden units at most for a Scout-sized problem. A small network trains in a second on a laptop and its failures are easy to read. Large networks are for large datasets, and robots rarely have as much data as the textbook examples.

Several seeds

Different random initialisations reach different functions with different errors (the H = 8 spread above). Train several and report the spread, as in lesson M1-T5.

Common misconceptions

Engineering takeaways

1. A hidden layer is a matrix product followed by a bend; stacking them gives a flexible curve.
2. Backpropagation computes ∂L/∂W2 = δ2 hᵀ and ∂L/∂W1 = δ1 xᵀ with δ1 = (W2ᵀ δ2) ⊙ (1 − h²); every gradient is a product of local slopes.
3. A 4 → 8 → 2 network has 58 parameters; count them against the amount of data.
4. Check gradients with a central difference; a correct implementation shows a relative error well below 1e−6 (2.3e−11 in the worked example).
5. Capacity let four tanh units follow a curve a line could explain only 6% of, but fitting the training data is not yet generalising (lesson M2-T5).

References and further reading

The original backpropagation paper, an approximation theorem, the two initialisation papers and a standard textbook.

• D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors”, Nature 323, 1986.
• G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals and Systems 2, 1989.
• X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTATS, 2010.
• K. He, X. Zhang, S. Ren and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, ICCV, 2015.
• I. Goodfellow, Y. Bengio and A. Courville, “Deep Learning”, MIT Press, 2016 (chapters 6 and 8).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: a model that fits the training data is not yet a model that works. Lesson M2-T5 separates fitting from generalising, with a degree sweep, regularisation and early stopping.

Related content

Nothing below is required to finish this lesson.

• Before
  M2-T3: Loss Functions and Gradient Descent

  Write the squared-error loss and its gradient, and run gradient descent with a sensible step size.

• Lab
  M2-L2: Train a Tiny Neural Network

  A linear model cannot follow a curved corridor: write the forward pass and backward pass of a two-layer network.