2.3 — CLIP and Vision-Language Alignment

Learning objectives

After this lesson, you should be able to:

• Explain what a shared image-text space is.
• Compute a table of cosine similarities.
• Describe the contrastive objective: matching pairs up, others down.
• Explain the difference between a softmax and a pairwise sigmoid loss.

The one-sentence idea

Train two encoders so that a picture and its caption land close together.

Comparing a picture with a sentence becomes comparing two vectors.

The idea: put pictures and words in one space

Imagine two translators. One reads a photo and writes down a list of numbers. The other reads a caption and writes down a list of numbers too. Training makes them agree: a photo of a red cube and the caption “a red cube” should get nearly the same numbers, and a photo of a blue ball should get different ones. After training, comparing a picture with a sentence is just comparing two lists of numbers.

That is CLIP. It learns image representations from raw text paired with images, using the simple task of predicting which caption goes with which image; natural language can then refer to the learned visual concepts, which allows zero-shot transfer to new tasks.^[1] It was pre-trained on 400 million image-text pairs collected from the internet,^[1] and its zero-shot accuracy on ImageNet matched the original ResNet-50 without using any of that dataset’s 1.28 million training examples.^[1]

Comparing by angle: cosine similarity

“Agree” needs a number. The usual choice is cosine similarity: the cosine of the angle between two vectors. It is 1 when they point the same way, 0 when they are unrelated and −1 when they are opposite. For a batch, compare every image with every caption to get a table of scores. Training pushes the matching pairs, the diagonal, up and everything else down.

import numpy as np

def normalise(v):
    return v / np.linalg.norm(v, axis=1, keepdims=True)

images = np.array([[0.9, 0.1, 0.0],       # pretend outputs of the image encoder
                   [0.1, 0.8, 0.2],
                   [0.0, 0.2, 0.9]])
texts  = np.array([[1.0, 0.0, 0.1],       # pretend outputs of the text encoder
                   [0.0, 1.0, 0.1],
                   [0.1, 0.1, 1.0]])
scores = normalise(images) @ normalise(texts).T
print(np.round(scores, 2))
print("best caption for each image:", scores.argmax(axis=1).tolist())

[[0.99 0.11 0.11]
 [0.14 0.98 0.35]
 [0.1  0.31 0.99]]
best caption for each image: [0, 1, 2]

Toy model. The vectors are made up. Real encoders output hundreds of numbers each.

The best score in each row is on the diagonal, so every image finds its own caption. Real training is about reaching this state from random starting vectors.

Training one: a tiny contrastive learner

Below is a very small version of the training loop. The “images” are noisy 8-number features of four things (red cube, blue cube, red ball, blue ball), and the “captions” are one-hot ids. Each encoder is just a matrix. Each step takes one picture of every thing, scores all pairs, and nudges the matrices so that matching pairs score higher than non-matching ones, in both directions (image-to-text and text-to-image). Nothing here is hidden; the gradients are written out.

import numpy as np
rng = np.random.default_rng(0)

names = ["red cube", "blue cube", "red ball", "blue ball"]
base = np.array([[1,0,1,0], [0,1,1,0], [1,0,0,1], [0,1,0,1]], float)   # colour and shape facts
mix = rng.normal(size=(4, 8))
def images(idx):                                   # noisy "photos" of each thing
    return base[idx] @ mix + 0.3 * rng.normal(size=(len(idx), 8))
def texts(idx):                                    # captions as one-hot ids
    return np.eye(4)[idx]

def unit(x):  return x / np.linalg.norm(x, axis=1, keepdims=True)
def softmax(x):
    e = np.exp(x - x.max(1, keepdims=True)); return e / e.sum(1, keepdims=True)

Wi = rng.normal(size=(8, 3)) * 0.3                 # image encoder (a matrix)
Wt = rng.normal(size=(4, 3)) * 0.3                 # text encoder (a matrix)
temp = 0.1

def accuracy():                                    # does each fresh photo pick its own caption?
    idx = np.arange(4).repeat(50)
    S = unit(images(idx) @ Wi) @ unit(texts(np.arange(4)) @ Wt).T
    return float((S.argmax(1) == idx).mean())

print("accuracy before training:", round(accuracy(), 2))
for step in range(300):
    idx = rng.permutation(4)                       # one photo of each thing, shuffled
    X, T = images(idx), texts(idx)
    A, B = X @ Wi, T @ Wt
    An, Bn = unit(A), unit(B)
    S = An @ Bn.T / temp
    P, Q = softmax(S), softmax(S.T)                # image->text and text->image
    loss = -(np.log(np.diag(P)).mean() + np.log(np.diag(Q)).mean()) / 2
    I = np.eye(4)
    dS = ((P - I) + (Q - I).T) / (2 * 4) / temp
    def back(dn, x, xn):                           # gradient through the normalisation
        return (dn - (dn * xn).sum(1, keepdims=True) * xn) / np.linalg.norm(x, axis=1, keepdims=True)
    Wi -= 0.5 * X.T @ back(dS @ Bn, A, An)
    Wt -= 0.5 * T.T @ back(dS.T @ An, B, Bn)
    if step in (0, 100, 299):
        print("step", step, "loss", round(float(loss), 3))
print("accuracy after training:", round(accuracy(), 2))

accuracy before training: 0.07
step 0 loss 7.521
step 100 loss 0.003
step 299 loss 0.001
accuracy after training: 1.0

Toy model. Two matrices, four concepts and a made-up photo model. CLIP trains large neural networks on hundreds of millions of real image-text pairs, but the objective has the same shape: match the right pairs inside each batch.

The loss falls and the accuracy goes from 0.07 (worse than the 0.25 you would get by guessing) to 1.0 on this easy problem. “Zero-shot” in CLIP means the same kind of test on a new task: describe the choices in words, embed the words, and pick the closest. No extra training on that task is needed.

A variation: SigLIP’s sigmoid loss

The loss above compares each image against all captions in the batch at once (softmax). SigLIP proposes a pairwise sigmoid loss that operates on individual image-text pairs, so it needs no global view of all the pairwise similarities for normalisation, unlike the softmax-normalised contrastive loss in CLIP-style training.^[2] In effect each pair gets its own question: “do these two match, yes or no?”. SigLIP-pretrained encoders appear inside the VLAs you will study, as OpenVLA fuses SigLIP and DINOv2 features^[3] and PaliGemma uses a SigLIP image encoder.^[4]

import numpy as np

scores = np.array([[ 0.9, 0.1, 0.0],
                   [ 0.1, 0.8, 0.2],
                   [ 0.0, 0.2, 0.9]])
t, b = 10.0, -5.0                                    # a scale and an offset (learnable in real models)
labels = 2 * np.eye(3) - 1                           # +1 for matching pairs, -1 for the others

pair_loss = np.log1p(np.exp(-labels * (t * scores + b)))     # each pair judged on its own
print("sigmoid-style loss, per pair:")
print(np.round(pair_loss, 2))
print("mean:", round(float(pair_loss.mean()), 3))

sigmoid-style loss, per pair:
[[0.02 0.02 0.01]
 [0.02 0.05 0.05]
 [0.01 0.05 0.02]]
mean: 0.026

Toy model. A simple pairwise logistic loss written for illustration. The numbers, the scale t and the offset b are made up; this shows the idea of judging each pair independently, not the SigLIP implementation.

The matching pairs (the diagonal) have small losses, the non-matching ones are also small, and no pair needs to know about the others. The paper reports that this allows further scaling of the batch size and also performs better at smaller batch sizes.^[2]

Why this matters for robots

An instruction like “pick up the red block” must be linked to what the camera sees. A model whose image and text vectors already live in one space makes that link easier to learn. That is why vision encoders and language models in VLAs are usually pre-trained this way, and the next lesson shows how the two are joined.

Common misconceptions

Engineering takeaways

1. Two encoders, one shared space.
2. Cosine similarity compares by angle.
3. Contrastive training raises matching pairs and lowers the rest.
4. Zero-shot = describe the options in words and pick the closest.
5. SigLIP judges each pair independently.

CLIP makes a picture and its caption point the same way.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: how vision encoders, language models and embeddings are fused into one sequence.

Back to the VLA roadmap