Skip to the lesson
FixTheRobot Explore Robotics / Perception / P9 Modern Robotics Perception and Capstone / P9-T2

Perception · Theory · P9

P9-T2 — Vision Transformers and Foundation Models

Attention lets every part of the image look at every other part. Pretraining on enormous data makes the result reusable, and also opaque.

  • Lesson 2 of 6
  • Module 9: Modern Robotics Perception and Capstone
  • Theory
  • About 8 minutes to read

Many current perception models replace convolution with attention: they cut the image into patches, treat the patches as a sequence of tokens and let every token gather information from every other. Trained on very large datasets, such models serve as foundation models: general-purpose feature extractors or promptable segmenters that can be adapted to a robot’s task with little extra data. This lesson sets out the mechanism, its cost, the way foundation models are used, and what they do and do not change for a perception system. Attention itself is treated in depth in the VLA track; here we take the vision-specific parts and the consequences for a robot.

Learning objectives

After this lesson, you should be able to:

  • Describe how an image becomes tokens and what self-attention computes.
  • Compute the number of tokens and the size of the attention matrix for a patch size, and the effect of changing it.
  • Distinguish training from scratch, fine-tuning, linear probing and zero-shot use of a pretrained model.
  • State what a foundation model adds to a robot’s perception and which risks it brings.

The idea: global context from the first layer

A convolution looks at a small neighbourhood; to see a whole object a network needs many layers (Lesson P9-T1). Attention does not work that way. The image is cut into square patches, each patch is turned into a vector (a token), the position of the patch is added, and in every layer each token computes a weighted combination of all tokens, with weights learned from how relevant each other token is to it. Context from the other side of the image is available at once.

A vision transformerAn image cut into a four by four grid of patches. The patches become a row of tokens with position information added, pass through attention layers where every token can look at every other token, and the output tokens feed a head for classes, boxes or masks.image as patches tokens (patch embeddings + position) self-attention: every token looks at every other Headclasses, boxes, masks Cost of attention grows with the square of the number of tokens.
A vision transformer: patches become tokens, attention lets every token read every other, and a head produces the output.

Why it needs data

A convolution carries a built-in assumption: patterns are local and the same everywhere. Attention carries almost none, so it must learn such structure from data. This is why vision transformers did not beat convolutional networks on small datasets, and why they do on very large ones. The more data, the less the built-in assumption matters.

Foundation models

A foundation model is trained once, on a very large and varied dataset, with an objective that does not need task labels (predicting masked patches, matching images with captions, or labelling many things at once), and then reused for many tasks. There are three ways to use one.

  1. Zero-shot. Use the model as it is. A promptable segmentation model returns a mask for a point or a box you give it; an image-text model scores a picture against names you supply (Lesson P9-T3).
  2. Linear probe. Freeze the model, take its features, and train only a small classifier or head on your labelled data. It needs little data and cannot break the features.
  3. Fine-tuning. Continue training all or part of the model on your data, usually with a small learning rate. It adapts best and needs the most data and care.

What they add to a robot

Robust general features, so that a small robot dataset goes a long way; masks and boxes for objects never seen in the robot’s training set, from a point or a box prompt, which is useful for labelling and for tasks with new objects; and a shared representation across tasks, which is what the vision-language-action models of the VLA track build on.

What they do not add

A guarantee. A foundation model is trained on web-scale data with no knowledge of your camera, your lighting or your safety requirements. It is bigger and slower than a task-specific network, harder to inspect, and its failures are less predictable, because its range of behaviour is wider. Its outputs still need alignment, uncertainty handling, monitoring and a fallback, like any other.

The mathematics: attention and its cost

Stack the N token vectors, each of dimension d, in a matrix X. Each layer computes queries, keys and values by learned linear maps, and the output of an attention head is

self-attention
Q=X WQQ=X\,W_{Q}
K=X WKK=X\,W_{K}
V=X WVV=X\,W_{V}
Attention⁡(X)=softmax⁡(Q K⊤dk) V\operatorname{Attention}(X)=\operatorname{softmax} (\frac{Q\,K^{\top}}{\sqrt{d_{k}}})\,V
Q, K, V
queries, keys and values: three learned projections of the tokens, each N × dk.
Q Kᵀ
an N × N matrix of scores: how strongly token i attends to token j.
√dk
a scaling that keeps the scores from growing with the dimension, so that the softmax does not saturate.
softmax
applied to each row, it turns the scores into weights that are positive and sum to one.
Attention(X)
for each token, the weighted average of the value vectors of all tokens.

Token count and cost. An image of H × W pixels cut into patches of size p × p gives

tokens and attention cost
N=(Hp)⋅(Wp)N=(\frac{H}{p})\cdot (\frac{W}{p})
attention matrix size=N2\text{attention matrix size}=N^{2}
cost∝N2⋅d\text{cost}\propto N^{2}\cdot d

Halving the patch size multiplies N by 4 and the attention matrix by 16. Attention cost is quadratic in the number of tokens, which is why high-resolution images are expensive for transformers, and why many variants limit attention to local windows or to a subset of tokens.

Position. Attention itself ignores the order of tokens. Position information is added to the token vectors (learned or sinusoidal) so that the model knows where each patch is, which matters for any task that outputs boxes or masks.

Worked examples: tokens, attention and the cost of resolution

Tokens. A 224 × 224 image with 16 × 16 patches has N = 14 × 14 = 196 tokens. The attention matrix has 196² = 38,416 entries per head per layer. With 8 × 8 patches the image has 28 × 28 = 784 tokens, and the matrix has 784² = 614,656 entries, 16 times more, for a finer view of the same picture.

A tiny attention step. Three tokens with scalar scores for the first token against all three of 2, 0 and 0 (after scaling). The softmax weights are e2 / (e2 + 1 + 1) = 7.389 / 9.389 = 0.787, and 1 / 9.389 = 0.107 for each of the other two. The first token’s output is 0.787 times the first value plus 0.107 times each of the others: it mostly keeps itself, with a little context from the rest.

High resolution. A robot camera at 1280 × 720 with 16 × 16 patches has 80 × 45 = 3,600 tokens; the attention matrix has 12.96 million entries per head per layer, 337 times the 224 × 224 case. This is why robot deployments often run a transformer at reduced resolution, on crops, or with windowed attention, and why the cost must be checked on the target device (Lesson P9-T5).

Choosing how to adapt.

SituationUseWhy
A new object appears; no labelled data yetzero-shot, promptable modelIt needs only a prompt (a box or a click), no training.
100 labelled images of your partslinear probe on frozen featuresLittle data; training only a small head cannot damage the general features.
5,000 labelled images from your robot, a fixed taskfine-tune the modelEnough data to adapt the features to your camera and conditions.
A safety function that needs a guaranteed response timea smaller task-specific network, with the foundation model as a helper offlineLatency and failure behaviour must be predictable.

What the capstone shows. In Lab P9-L2 the detections feed a pipeline that is indifferent to how they were made. If a promptable model were producing masks, the same steps would apply: depth, lifting, extrinsic, filters, report. The model would change the accuracy of the first step; it would not remove the need for the others.

Engineering implications and limitations

Use foundation models where they save labelling. Prompted masks and boxes can produce labels that humans then correct, cutting annotation time. Keep the human check: the labels inherit the model’s errors.

Do not put an unmonitored large model in a safety loop. Use it where its latency and failure modes are acceptable, with the monitoring, fallback and checks of Lesson P9-T6.

Measure the cost on the target. Attention memory and time grow quickly with resolution. Test the real latency and the worst case at the resolution you will use.

Check domain fit. A web-trained model has seen few images from a robot’s low viewpoint, an industrial setting or an infrared camera. Evaluate it by condition on your data before relying on it (Lesson P6-T6).

Distil for deployment. A large model can teach a small one: train a small, fast network to reproduce its outputs on your data. The small network is cheaper and easier to validate.

Record the model version. Foundation models are updated. Pin the exact weights and test again on every change, as for any component.

Stay honest about this track. No transformer is run in the labs. Everything here describes how such models work and how to use them; the labs use simulated detections.

Common misconceptions

“Transformers replaced convolutions everywhere.”

Both are used, and hybrids are common. Convolutions remain efficient and data-efficient for many robot deployments.

“A foundation model works on any image.”

It has seen a lot, but not your camera, viewpoint or conditions. It needs evaluation on your own data like any model.

“Zero-shot means no failure analysis needed.”

It means no training. Failures still exist, are harder to predict, and must be measured.

“Bigger is always better.”

Bigger models cost latency and memory (quadratically in tokens for attention) and are harder to validate; the smallest model that meets the requirement is usually the right one.

Engineering takeaways

  1. A vision transformer cuts the image into patches, treats them as tokens and applies self-attention: softmax(QKᵀ/√d)V.
  2. Tokens N = (H/p)(W/p); the attention matrix has N² entries: 196 tokens → 38,416; with half the patch size 784 tokens → 614,656 (16 times).
  3. Foundation models are pretrained on large data and used zero-shot, by linear probe, or by fine-tuning, depending on the data you have.
  4. They add general features and promptable masks; they do not add guarantees, and cost latency, memory and opacity.
  5. Evaluate them on your own conditions, pin the version and keep the checks of the rest of the pipeline.

References and further reading

Papers on transformers, vision transformers and foundation models. The token counts and attention-matrix sizes in the examples were computed from the formulas.

  • Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.
  • Dosovitskiy, A. et al. (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR.
  • Kirillov, A. et al. (2023). Segment Anything. ICCV.
  • Oquab, M. et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research.
  • Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A 512 × 512 image is cut into 32 × 32 patches. How many tokens are there, and how many entries does the attention matrix have?

Explanation

N = 16 × 16 = 256 tokens; the attention matrix has 256² = 65,536 entries.

Question 2

You halve the patch size on the same image. By what factors do the number of tokens and the attention matrix grow?

Explanation

Tokens ×4 (twice as many patches in each direction); the attention matrix ×16 (4²).

Question 3

You have 60 labelled images from your robot and a promptable foundation model. Which way of using it do you try first, and why?

Explanation

First zero-shot with prompts (no training), and a linear probe on frozen features if a classifier is needed: with 60 images, fine-tuning risks overfitting and damaging the general features.

What’s next

Next: models that connect images with language, open-vocabulary detection and vision-language models.

Next lesson

P9-T3 — Open-Vocabulary Detection and Vision-Language Models

Explain how language and vision are linked and what open-vocabulary detection can and cannot do.

Nothing below is required to finish this lesson.