P9-T1 — CNNs and Modern Visual Perception

Learning objectives

After this lesson, you should be able to:

• Describe a convolution and why weight sharing suits images.
• Compute the number of parameters and the multiply-accumulate count of a convolutional layer.
• Compute the receptive field of a stack of convolutions.
• Describe the backbone-and-head structure of a detector and what limits its behaviour.

The idea: the same small detector, everywhere in the image

An edge looks the same wherever it appears. A convolution exploits this: a small filter, say 3 × 3 pixels with its own weights, slides over the image, and at each position computes a weighted sum of the pixels under it. The same nine weights are used everywhere, so the layer needs very few parameters and responds to a pattern wherever it occurs. Lesson P2-T3 introduced convolution with hand-made kernels (blur, edges); a network learns its kernels from examples.

Layers and features

A layer applies many filters at once and produces one feature map (channel) per filter. Between layers a nonlinearity (commonly ReLU, which sets negatives to zero) lets the stack represent more than a single linear filter, and pooling or strides reduce the resolution. Stacking many layers builds a hierarchy: first edges and colours, then textures and corners, then parts, then whole objects. Nobody designs the features; they emerge from training on labelled images by gradient descent, as in the learning chapters of the VLA track.

Backbone and head

A detector is usually built in two parts. The backbone is a stack of convolutional layers that turns the image into feature maps at several resolutions; it is often pretrained on a large classification dataset and reused. The head is a smaller network on top of the features that predicts, at each location, a box offset, a score and class probabilities, the anchor-based output decoded in Lesson P6-T2. Segmentation heads produce a class per pixel instead.

Training data decide the behaviour

The weights are fitted to a training set (Lesson P6-T4). Augmentation (random crops, flips, colour changes, noise) enlarges the variety the network sees and makes it less sensitive to those changes. Whatever is not in the data, such as a new object type, a different camera or a weather the training set lacked, is outside what the network has been shown to handle (Lesson P6-T6).

What a network does not give you

A trained network is a function, not a specification. It has no guarantee outside its training distribution, its scores are not calibrated probabilities by default (Lesson P6-T2), it can be confidently wrong, and a tiny, carefully chosen change in the image, or an ordinary change in lighting, can change its output. These are not reasons to avoid it. They are reasons to wrap it with the checks of the rest of the track: alignment, uncertainty, conflict detection, monitoring and a fallback.

The mathematics: convolution, parameters, compute and receptive field

A convolutional layer with a k × k kernel, C_in input channels and C_out output channels computes, at each output position and for each output channel o,

x[c, i, j]

the input value at channel c and position (i, j).

W[o, c, u, v]

the learned weights: one k × k kernel for every pair of an input and an output channel.

b[o]

a learned bias for each output channel.

(u, v)

the offsets within the kernel window, from 0 to k − 1 (or centred).

Parameters. The number of weights is independent of the image size:

Compute. Each output value needs k² C_in multiply-accumulates (MACs), so for an output of size H × W

One MAC is two floating-point operations (a multiplication and an addition). Compute scales with the image area; parameters do not.

Receptive field. The receptive field is the region of the input that can influence one output value. For stride-1 layers with kernel k, each layer adds k − 1 to it:

With strides or pooling, later layers see further per layer: the increment is multiplied by the product of the earlier strides.

Worked examples: counting parameters, compute and receptive field

One layer. A 3 × 3 convolution with 64 input and 128 output channels:

On a 56 × 56 feature map (stride 1, same size output):

That is about 0.46 GFLOPs for this single layer. A network has dozens of such layers; a mid-size detector needs several to tens of GFLOPs per frame.

Why convolution. A fully connected layer from the same input (56 × 56 × 64 = 200,704 values) to the same output (56 × 56 × 128 = 401,408 values) would need 200,704 × 401,408 ≈ 8.1 × 10¹⁰ weights, about a million times more than the 73,856 above, and it would not share what it learns between positions. Weight sharing is what makes learning from images practical.

Receptive field. Three stacked 3 × 3 stride-1 layers have a receptive field of 1 + 3 × 2 = 7 pixels, the same as one 7 × 7 kernel, with 3 × 9 = 27 weights per channel pair instead of 49. With a stride-2 step after the first layer, the second and third layers see twice as far per layer: 1 + 2 + 2·2 + 2·2 = 11 pixels. Deep networks with several downsampling steps reach receptive fields of hundreds of pixels, wide enough to see whole objects.

Reading a model’s cost. A detector of 4 GFLOPs per frame at 30 Hz needs 120 GFLOP/s. A device with a peak of 1 TFLOP/s but a realistic 30% efficiency delivers 300 GFLOP/s, so this model uses 40% of the achieved throughput, leaving room for the rest of the pipeline. A model of 20 GFLOPs would not fit (600 GFLOP/s required) and would run at about 15 Hz (Lesson P9-T5).

The detector in the capstone. In Lab P9-L2 the detections are produced by a simulator with a stated error model: boxes with scores, loose around the object, with false alarms on the floor. They behave like the output of a head on top of a backbone, so everything you build around them (depth, frames, filters, reports) applies unchanged to a real network’s output. The labs do not contain a trained network and say nothing about the accuracy of any.

Engineering implications and limitations

Start from a pretrained backbone. Training from scratch needs far more data than most robotics projects have. Take a backbone pretrained on a large dataset and fine-tune it on your data.

Match the model to the compute. Count the parameters and the MACs, check them against the device, and measure the actual latency on it, not on a workstation (Lesson P9-T5).

Collect data from your own robot. The camera, the height, the lighting and the objects of the deployment decide whether a public dataset’s model works. Label a representative set and keep it for testing (Lesson P6-T4).

Use augmentation to cover what you expect. Crops, brightness and contrast, blur and noise cover predictable variation. They cannot cover what you did not think of.

Keep the geometry outside the network. Calibration, frames, timing and tracking are better done by explicit, testable code (Modules 3, 7 and 8) than learned implicitly.

Treat the network’s output as a claim. Check its scores against reality by condition, calibrate or threshold them with data (Lesson P6-T2), and design for the case when it is wrong.

Know the licence and the provenance. A pretrained model carries the terms and the biases of its training data; record where it came from.

Common misconceptions

Engineering takeaways

1. A convolution slides a small learned kernel over the image; weight sharing makes the parameter count independent of the image size.
2. Parameters = k²·C_in·C_out + C_out; MACs = H·W·k²·C_in·C_out (73,856 parameters and 0.23 G MACs for the 64 → 128 layer on 56 × 56).
3. The receptive field of n stride-1 3 × 3 layers is 1 + 2n; strides make later layers see further.
4. A detector is a backbone (features) plus a head (boxes, scores, classes); behaviour is set by the training data.
5. A trained network has no guarantee outside its data: wrap it with alignment, uncertainty, conflict checks, monitoring and a fallback.

References and further reading

Foundational papers and a textbook. The layer sizes and device figures in the examples are illustrative; no trained network is run in this track.

• LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE.
• Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS.
• He, K., Zhang, X., Ren, S. and Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR.
• Ren, S., He, K., Girshick, R. and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NeurIPS.
• Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning. MIT Press: convolutional networks.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: transformers and foundation models, which replace convolution with attention and train on far more data.

Related content

Nothing below is required to finish this lesson.

• Before
  P8-T6: Scene Representations for Downstream Robotics Tasks

  Choose a scene representation for navigation, manipulation and interaction.

• Lab
  P9-L2: Capstone: Build an End-to-End Perception Pipeline

  From simulated camera and range observations to a structured obstacle report for a downstream task, with diagnosis of injected failures.