M4.2 — Basic Computer Vision Concepts

Learning objectives

After this lesson, you should be able to:

• Explain what a visual feature is.
• Apply a kernel to a small image by hand and by code.
• Compute the output size of a convolution from padding and stride.
• Explain pooling and receptive fields.
• Name the common vision tasks and what each outputs.
• Compute the intersection over union of two boxes.

The one-sentence idea

A convolutional layer slides small learned detectors across an image, and stacking layers turns edges into shapes and shapes into objects.

Same small kernel, every position.

What a visual feature is

A raw pixel tells you very little: one pixel is a single brightness or colour. What matters in an image is structure: edges where brightness changes sharply, corners, textures, repeated patterns, and, at a higher level, shapes and objects. These pieces of structure are called features.

Before deep learning, engineers designed feature detectors by hand: an edge detector, a corner detector, and so on. A convolutional neural network (CNN) learns its detectors from data instead. Both rest on the same mathematical operation, convolution, which is worth understanding because it is the first layer of most vision models.

Convolution: sliding a small detector

A kernel (or filter) is a small array of weights. Convolution slides it across the input. At each position it multiplies the kernel with the patch of input under it, entry by entry, and sums the result into one output number. A kernel that responds to a certain pattern gives large outputs wherever that pattern appears. (Strictly, deep-learning libraries compute cross-correlation, which does not flip the kernel, but everyone calls it convolution.)

In one dimension, a kernel [-1, 0, 1] subtracts the value on the left from the value on the right, so it responds to a jump in the signal:

import numpy as np

signal = np.array([0, 0, 0, 10, 10, 10])
kernel = np.array([-1, 0, 1])
print(np.correlate(signal, kernel, mode="valid"))

[ 0 10 10  0]

The output is 0 over the flat regions and 10 where the signal jumps, so it has found the edge. In two dimensions the kernel is a small grid. The kernel below, called a Sobel kernel, responds to vertical edges, where brightness changes from left to right:

def conv2d(image, kernel):
    kh, kw = kernel.shape
    out = np.zeros((image.shape[0] - kh + 1, image.shape[1] - kw + 1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            out[i, j] = (image[i:i + kh, j:j + kw] * kernel).sum()   # multiply and add
    return out

image = np.array([[0, 0, 10, 10, 10]] * 5, dtype=float)    # dark on the left, bright on the right
sobel_x = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]], dtype=float)
print(image.shape, "->", conv2d(image, sobel_x).shape)
print(conv2d(image, sobel_x))

(5, 5) -> (3, 3)
[[40. 40.  0.]
 [40. 40.  0.]
 [40. 40.  0.]]

The output is large (40) in the columns whose window crosses the dark-to-bright boundary and zero over the flat bright region. A 5 × 5 image with a 3 × 3 kernel gives a 3 × 3 output, because the kernel needs a full window: the output shrinks by kernel size − 1.

Learned kernels, padding, stride and pooling

A convolutional layer holds many kernels and learns their weights by training (M2.4). Each kernel produces one feature map, and a layer with 16 kernels produces 16 maps, which become the channels of the next layer’s input. Early layers tend to learn edge-like detectors. Later layers combine them into parts and objects.

Two settings control the output size. Padding adds a border of zeros so the output does not shrink. Stride moves the kernel more than one pixel at a time, which reduces the resolution. The output length along one dimension is ⌊(n + 2p − k) / s⌋ + 1 for input size n, kernel k, padding p and stride s:

def out_size(n, k, p=0, s=1):
    return (n + 2 * p - k) // s + 1

print(out_size(32, 3), out_size(32, 3, p=1), out_size(32, 3, p=1, s=2))

30 32 16

A 3 × 3 kernel on a 32-pixel row gives 30 outputs, padding of 1 keeps all 32, and a stride of 2 halves it to 16. Pooling is another way to reduce resolution: max-pooling keeps the largest value in each small block, which keeps the strongest response and makes the result less sensitive to small shifts.

fm = np.array([[1, 3, 2, 0],
               [4, 2, 1, 1],
               [0, 1, 5, 2],
               [2, 0, 3, 4]])
pooled = fm.reshape(2, 2, 2, 2).max(axis=(1, 3))     # split into 2 x 2 blocks, take each maximum
print(pooled)

[[4 2]
 [2 5]]

Each 2 × 2 block is replaced by its largest entry, so the 4 × 4 map becomes 2 × 2. Stacking layers like this builds up the receptive field: the region of the original image that can affect one output value. With stride 1, each 3 × 3 layer adds 2 to it:

rf = 1
for layer in range(1, 5):
    rf += 2                       # each 3x3 layer, stride 1, widens the view by 2
    print(layer, rf)

1 3
2 5
3 7
4 9

Four stacked 3 × 3 layers see a 9 × 9 window. Deep networks reach whole images by stacking many layers and by using stride and pooling to shrink the maps along the way. PyTorch provides all of this ready-made, and it agrees with the hand-written version:

import torch
from torch import nn

conv = nn.Conv2d(1, 1, kernel_size=3, bias=False)
with torch.no_grad():
    conv.weight[:] = torch.tensor(sobel_x, dtype=torch.float32).reshape(1, 1, 3, 3)
x = torch.tensor(image, dtype=torch.float32).reshape(1, 1, 5, 5)     # (batch, channels, H, W)
print(torch.allclose(conv(x)[0, 0], torch.tensor(conv2d(image, sobel_x), dtype=torch.float32)))

layer = nn.Conv2d(3, 16, kernel_size=3)      # 3 input channels, 16 kernels
print(sum(p.numel() for p in layer.parameters()))

True
448

The layer’s 448 parameters are 16 kernels × (3 channels × 3 × 3 weights) = 432, plus one bias per kernel (16). Note how few parameters this is for a layer that looks at the whole image: the same small kernel is reused at every position.

Common vision tasks

Vision problems differ mostly in what the model outputs:

Detection is judged by how well a predicted box overlaps the true one, measured by intersection over union (IoU): the area of the overlap divided by the area of the union. It is 1 for a perfect match and 0 for no overlap.

def iou(a, b):
    """Boxes as (x1, y1, x2, y2)."""
    ix = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    iy = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ix * iy
    union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter
    return inter / union

print(round(iou((0, 0, 4, 4), (2, 2, 6, 6)), 3))
print(iou((0, 0, 4, 4), (0, 0, 4, 4)), iou((0, 0, 1, 1), (5, 5, 6, 6)))

0.143
1.0 0.0

Two 4 × 4 boxes that overlap in a 2 × 2 corner have an overlap of 4 and a union of 16 + 16 − 4 = 28, so the IoU is 4/28 ≈ 0.143. For a robot, these outputs feed directly into action: a detected box or a segmentation mask says where to reach, and a depth map says how far. A VLA model usually learns such abilities internally instead of exposing them as separate outputs, but these are the capabilities that its vision encoder has to supply.

Common misconceptions

Engineering takeaways

1. Features are structure in an image: edges, corners, textures, shapes.
2. Convolution slides a small kernel over the image, multiplying and summing at each position.
3. A convolutional layer has many learned kernels and produces one feature map per kernel.
4. Output size is ⌊(n + 2p − k) / s⌋ + 1.
5. Pooling and stride reduce resolution. Stacking layers grows the receptive field.
6. Weight sharing keeps convolutional layers small: 448 parameters for 16 kernels over 3 channels.
7. Tasks differ by output: a label, boxes, a per-pixel map, depth or pose.
8. IoU measures how well two boxes overlap.

A vision network is a stack of detectors, each looking at a small patch and passing its findings upward.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You now know how a vision network finds structure. Next is how whole encoders turn an image into compact features that other models can use.

Back to the VLA roadmap prerequisites