M4.5 — CLIP and Vision-Language Alignment

Learning objectives

After this lesson, you should be able to:

• Explain what a shared image-text embedding space is for.
• Describe CLIP’s two encoders and the similarity table.
• Explain the contrastive loss and why batch size matters.
• Explain zero-shot classification with text prompts.
• Run and interpret a toy two-tower training example.
• Name the main limits of CLIP-style models for robotics.

The one-sentence idea

Train an image encoder and a text encoder so that each picture’s vector is closest to the vector of its own caption among everything in the batch.

Matching pairs close, everything else far.

The goal: one space for pictures and words

A vision encoder (M4.3) maps images to vectors, and a language model maps text to vectors. Those two vector spaces are unrelated: the vector for a photo of a cup and the vector for the word “cup” point in arbitrary directions. Vision-language alignment trains the two encoders together so that a picture and a description of it land close together in one shared space. Once that holds, comparing an image with a sentence is just a similarity score (M3.2), and a model that has to relate what it sees to what it is told has a common language to work in.

CLIP (Contrastive Language-Image Pre-training, from OpenAI in 2021) is the model that made this approach standard. It was trained on a very large collection of image-and-caption pairs collected from the web, which is exactly the kind of cheap, abundant supervision that M3.5 described.

Two encoders and a similarity table

CLIP has an image encoder and a text encoder, one “tower” each, and each produces one vector per input. Both vectors are normalised to length 1, so the dot product of an image vector and a text vector is their cosine similarity. For a batch of n image-caption pairs the model computes the full n × n table of similarities between every image and every caption. The matching pairs lie on the diagonal:

import numpy as np

def unit(x):
    return x / np.linalg.norm(x, axis=1, keepdims=True)

images = unit(np.array([[1.0, 0.2, 0.0],      # photo of a cup
                        [0.1, 1.0, 0.1],      # photo of a bowl
                        [0.0, 0.2, 1.0]]))    # photo of a wrench
texts = unit(np.array([[0.9, 0.1, 0.1],       # "a cup"
                       [0.0, 0.9, 0.2],       # "a bowl"
                       [0.1, 0.1, 0.9]]))     # "a wrench"
sims = images @ texts.T
print(sims.round(2))
print(sims.argmax(axis=1))

[[0.99 0.19 0.13]
 [0.22 0.99 0.22]
 [0.13 0.4  0.99]]
[0 1 2]

Each image is most similar to its own caption (the diagonal values are all 0.99), and the arg-max of every row picks the right one. The vectors here are made by hand to show what a successful result looks like. Training is how real encoders get there.

The contrastive loss

Training treats each row of the table as a classification problem: given this image, which of the n captions is correct? The correct answer is the diagonal entry. The same is done for each column, asking which image matches each caption, and the two losses are averaged. The similarities are multiplied by a scale first (the inverse of a temperature, as in M4.4), so that softmax can become sharp. This pushes matching pairs together and, because every other pair in the batch acts as a wrong answer, pushes non-matching pairs apart:

def clip_loss(images, texts, scale=10.0):
    logits = scale * images @ texts.T                       # (n, n)
    def ce_rows(z):                                         # cross-entropy with the diagonal as the answer
        e = np.exp(z - z.max(axis=1, keepdims=True))
        p = e / e.sum(axis=1, keepdims=True)
        return -np.log(np.diag(p)).mean()
    return (ce_rows(logits) + ce_rows(logits.T)) / 2        # images -> texts, and texts -> images

print(round(clip_loss(images, texts), 3))
print(round(clip_loss(images, texts[[1, 2, 0]]), 3))        # captions given to the wrong images

0.001
8.103

With correct pairing the loss is close to zero, and with the captions shuffled it is large. A larger batch makes the task harder and more informative, because there are more wrong captions to tell apart, which is one reason CLIP-style models are trained with very large batches.

The code below trains two small encoders from scratch on a toy problem. There are eight “concepts”. A concept’s image is a noisy 16-number feature vector, and its caption is a different, 10-number vector, so the two inputs live in unrelated spaces, as real images and text do. Each tower is a single linear layer, and training uses the loss above:

import torch
from torch import nn
import torch.nn.functional as F

torch.manual_seed(0)
n = 8
img_proto, txt_proto = torch.randn(n, 16), torch.randn(n, 10)       # one fixed pattern per concept
img_enc, txt_enc = nn.Linear(16, 8), nn.Linear(10, 8)
opt = torch.optim.Adam(list(img_enc.parameters()) + list(txt_enc.parameters()), lr=0.05)

def batch():
    return img_proto + 0.1 * torch.randn(n, 16), txt_proto + 0.1 * torch.randn(n, 10)

def loss_and_acc(x, t, scale=10.0):
    zi, zt = F.normalize(img_enc(x), dim=1), F.normalize(txt_enc(t), dim=1)
    logits = scale * zi @ zt.T
    target = torch.arange(n)
    loss = (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2
    return loss, (logits.argmax(dim=1) == target).float().mean().item()

loss0, acc0 = loss_and_acc(*batch())
for step in range(200):
    loss, _ = loss_and_acc(*batch())
    opt.zero_grad(); loss.backward(); opt.step()
loss1, acc1 = loss_and_acc(*batch())
print(loss0.item() > 1.0, loss1.item() < 0.1)
print(acc1 == 1.0)

True True
True

The loss starts above 1 and ends below 0.1, and after training every image is matched to its own caption, using fresh noisy examples it never saw. The two towers began in unrelated spaces and were pulled into one shared space purely by the pairing signal. Nothing told the model what any concept “is”. That is the essence of CLIP.

Zero-shot recognition from text

Once the space is shared, you can classify an image without training a classifier. Write the candidate classes as short sentences, embed them with the text encoder, embed the image, and choose the class whose text vector is most similar. This is called zero-shot classification, because no labelled examples of those classes were used. It is also how a robot system can be pointed at a new object just by naming it.

classes = ["a cup", "a bowl", "a wrench"]
image = unit(np.array([[0.8, 0.3, 0.1]]))              # embedding of a new photo
scores = 100 * image @ texts.T                         # 100 is a typical learned scale
e = np.exp(scores - scores.max())
probs = (e / e.sum()).round(3)[0].tolist()
print(dict(zip(classes, probs)))

{'a cup': 1.0, 'a bowl': 0.0, 'a wrench': 0.0}

The photo is closest to the “cup” sentence, so that class gets nearly all the probability. A new object needs only a new sentence, not new training data.

What alignment does not give you

CLIP-style models are strong at what is in the picture, and notably weaker at finer structure. Treat the following as things to watch for rather than solved problems:

• Spatial relations and counting. A caption and its mirror image, “the cup is left of the bowl” and “the bowl is left of the cup”, can look alike to the model, and exact counts are unreliable. Both matter for manipulation.
• One vector per image. The standard output is a single global vector, which, as M4.3 showed, discards position. For robot control the local patch features are usually needed too.
• Web data. Captions on the web describe what people photograph and write about, so unusual viewpoints, wrist cameras and cluttered workbench scenes are underrepresented, and biases in the data carry into the model.
• Prompt sensitivity. The result can change with the wording, such as “a cup” versus “a photo of a cup”.

Despite these limits, aligned encoders are a core building block of VLA systems. They give a vision-language model its starting understanding of how words and images relate, which fine-tuning on robot data (M3.5) then adapts. Later roadmap phases look at how particular VLAs use them. Variants of the idea, such as models trained with a different loss, are common, but the shared-space principle is the same.

Common misconceptions

Engineering takeaways

1. Alignment trains an image encoder and a text encoder into one shared vector space.
2. Vectors are normalised, so the dot product is cosine similarity.
3. A batch gives an n × n table in which matching pairs sit on the diagonal.
4. The symmetric contrastive loss is cross-entropy over the rows and over the columns.
5. More non-matching pairs in a batch make the task, and the signal, stronger.
6. Zero-shot classification compares an image with text prompts for each class.
7. The standard output is one global vector, which loses position.
8. Spatial relations and counting are weak points.

Once words and pictures share a space, relating them is a matter of measuring distance.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You have now seen vision encoders, language models and alignment. The last lesson of this module covers how modalities are combined and how words are tied to specific things in the scene.

Back to the VLA roadmap prerequisites