M3.1 — Neural Network Fundamentals

Learning objectives

After this lesson, you should be able to:

• Explain why a network needs a nonlinear activation, using XOR.
• Name common activation functions and where each is used.
• Compute softmax and explain what it produces.
• Count a network’s parameters from its layer sizes.
• State what the universal approximation theorem does and does not say.
• Describe residual connections, normalisation and regularisation at a high level.

The one-sentence idea

A neural network is stacked linear layers with nonlinear activations between them, and its size and depth set how complicated a function it can represent.

Without the nonlinearity, depth adds nothing.

Where M2.4 left off

M2.4 showed how a small network computes an output and how backpropagation trains it. This lesson steps back to ask a different question: what can such a network represent at all, and what are the design choices (depth, width, activation, output layer) that decide the answer? Those choices are the vocabulary used to describe every model in the rest of the roadmap.

Why the activation function is essential

A stack of linear layers collapses into one linear layer (M1.1), and a single linear layer can only draw straight boundaries. The classic example is XOR: output 1 when exactly one of two inputs is 1. No straight line separates the 1s from the 0s, and the best a linear model can do is predict 0.5 everywhere:

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
t = np.array([0.0, 1.0, 1.0, 0.0])
A = np.hstack([X, np.ones((4, 1))])                   # add a column of ones for the bias
coef, *_ = np.linalg.lstsq(A, t, rcond=None)          # best linear fit in the least-squares sense
print((A @ coef).round(2))

[0.5 0.5 0.5 0.5]

One hidden layer with a ReLU is enough. Here the weights are chosen by hand (no training) so that you can see how it works:

W1 = np.array([[1.0, 1.0],
               [1.0, 1.0]])
b1 = np.array([0.0, -1.0])
w2 = np.array([1.0, -2.0])

h = np.maximum(0, X @ W1.T + b1)       # hidden layer, one row per input
y = h @ w2
print(h)
print(y)

[[0. 0.]
 [1. 0.]
 [1. 0.]
 [2. 1.]]
[0. 1. 1. 0.]

The first hidden unit measures “how many inputs are on”, and the second switches on only when both are, and the output subtracts twice that second signal. The nonlinearity lets the network bend its decision rule. Without the ReLU the same weights would produce a straight-line result.

Common activation functions:

The output layer: softmax

When a network chooses among several options, such as the next word or the next action token, its last layer produces one raw score (a logit) per option, and softmax converts them into probabilities. It exponentiates each score and divides by the total, so larger scores get larger probabilities and the results add up to 1.

z = np.array([2.0, 1.0, 0.1])
p = np.exp(z) / np.exp(z).sum()
print(p.round(3), p.sum().round(3))

[0.659 0.242 0.099] 1.0

Softmax appears again in the next lessons: attention uses it to decide how much weight each position receives, and language models use it to pick the next token.

What a network can represent: depth, width and capacity

Width is the number of units in a layer and depth is the number of layers. Together with the sizes of the layers they determine the number of parameters, which is often called the model’s capacity:

def n_params(sizes):
    """Weights plus biases for a stack of linear layers."""
    return sum(a * b + b for a, b in zip(sizes[:-1], sizes[1:]))

print(n_params([10, 64, 64, 2]))
print(n_params([10, 128, 128, 2]))

4994
18178

Doubling the width roughly quadruples the parameters of the middle layers. A famous result, the universal approximation theorem, says that a network with one hidden layer and enough units can approximate any reasonable continuous function on a bounded region as closely as you like. The theorem says an approximation exists. It does not say gradient descent will find it, how many units are needed, or that the network will behave well on inputs it never saw.

A ReLU network is built from bent line segments, and you can watch the approximation improve as units are added. Below, a sum of shifted ReLUs approximates x² on 0 to 1, with 4 units and then 8:

def relu_fit(f, n, xs):
    knots = np.linspace(0, 1, n + 1)
    slopes = np.diff(f(knots)) / np.diff(knots)
    coef = np.concatenate([[slopes[0]], np.diff(slopes)])   # change of slope at each knot
    return f(0.0) + sum(c * np.maximum(0, xs - k) for c, k in zip(coef, knots[:-1]))

xs = np.linspace(0, 1, 1601)
for n in (4, 8):
    err = np.abs(relu_fit(lambda x: x ** 2, n, xs) - xs ** 2).max()
    print(n, round(err, 4))

4 0.0156
8 0.0039

With twice as many units the worst-case error falls to a quarter. Real networks learn the knots and slopes from data instead of being given them, and use many more dimensions, but the principle is the same: more units can follow more complicated shapes.

What matters in practice

Representing a function and learning it are different problems. Practical networks add a few ingredients that you will meet in the transformer lessons:

• Residual connections add a layer’s input to its output, so gradients can flow through very deep networks.
• Normalisation layers (such as LayerNorm) keep activations at a stable scale.
• Dropout and other regularisers reduce overfitting by making the network less able to memorise.

Depth tends to help networks build up features in stages, for example edges, then shapes, then objects in vision. Very large models also need very large datasets: capacity without enough data leads to memorisation, which is why M2.5 held out validation data. In PyTorch, a network like those above is a sequence of layers:

import torch
from torch import nn

net = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 2))
print(net(torch.ones(4)).shape)
print(sum(p.numel() for p in net.parameters()))

torch.Size([2])
114

The parameter count, 4×16 + 16 + 16×2 + 2 = 114, matches what the n_params formula gives for sizes [4, 16, 2].

Common misconceptions

Engineering takeaways

1. A stack of linear layers is still linear, so networks need nonlinear activations.
2. XOR cannot be solved by a linear model but can by a network with one ReLU hidden layer.
3. ReLU, GELU, tanh, sigmoid and softmax are the activations to recognise.
4. Softmax turns a vector of scores into probabilities that sum to 1.
5. Parameters are weights plus biases, and grow quickly with width.
6. Universal approximation says a good fit exists, not that it will be found.
7. Residual connections, normalisation and regularisation make large networks trainable.
8. Capacity needs enough data, or the network memorises.

Linear layers do the arithmetic, and the nonlinearity gives the network its shape.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You now know what a network is and what sets its capacity. Next is how text and other inputs become the vectors a network can use.

Back to the VLA roadmap prerequisites