2.4 — Vision Encoders, Language Models, Embeddings and Fusion

Learning objectives

After this lesson, you should be able to:

• Explain what an embedding and an embedding table are.
• Describe fusing image and text by putting them in one sequence.
• Explain what a causal mask does to that fusion.
• Describe how FiLM lets an instruction modulate image features.

The one-sentence idea

Give pictures and words the same kind of vectors, then let attention or modulation mix them.

Fusion is just letting the two kinds of vectors influence each other.

Three parts and a join

A vision-language model has three jobs. A vision encoder turns an image into vectors. A language model turns words into vectors and processes sequences of them. And a fusion step lets the two talk. LLaVA, for example, is an end-to-end trained model that connects a vision encoder with a large language model for general-purpose visual and language understanding.^[1] PaliGemma is made from a SigLIP image encoder and the Gemma-2B language model,^[2] and OpenVLA joins its visual encoder to a Llama 2 language model through a projector.^[3]

Before fusion, each side needs numbers. That is what embeddings are.

Embeddings: words as points

An embedding is a list of numbers that stands for a token. The model keeps a table with one row per token in its vocabulary, and looking a word up means picking its row. Training arranges the rows so that tokens used in similar ways get similar vectors. Here is the lookup mechanism with hand-made vectors:

import numpy as np

vocab = ["red", "blue", "cube", "ball"]
table = np.array([[0.9, 0.1, 0.0],       # red
                  [0.8, 0.2, 0.1],       # blue  (a colour, so close to red)
                  [0.0, 0.9, 0.2],       # cube
                  [0.1, 0.8, 0.3]])      # ball  (a shape, so close to cube)

def embed(word):
    return table[vocab.index(word)]

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print("red  vs blue:", round(cosine(embed("red"), embed("blue")), 2))
print("red  vs cube:", round(cosine(embed("red"), embed("cube")), 2))
print("cube vs ball:", round(cosine(embed("cube"), embed("ball")), 2))

red  vs blue: 0.98
red  vs cube: 0.11
cube vs ball: 0.98

Toy model. The vectors are written by hand to make the point. Real embeddings are learned and have hundreds or thousands of numbers.

Colours are closer to colours and shapes to shapes. In a real model nobody writes this by hand; it emerges from training on lots of text.

Fusion by one long sequence

The most common recipe in the models above is to turn image patches into vectors, project them to the language model’s width, and simply put them in the same sequence as the word vectors. PaLM-E feeds images, robot state estimates and text, interleaved, into a pretrained language model.^[4] Once everything is in one sequence, attention (lesson 2.1) does the fusion: each word can read the image tokens.

Language models usually read left to right, so a token can only attend to tokens before it (a “causal mask”). If the image tokens come first, the words can see the image but the image cannot see the words. The example checks exactly that:

import numpy as np
rng = np.random.default_rng(0)

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def causal_attention(X):
    n, d = X.shape
    scores = X @ X.T / np.sqrt(d)
    scores[np.triu_indices(n, k=1)] = -1e9           # a token may not look at later tokens
    return softmax(scores) @ X

n_img, n_txt, d = 4, 3, 8
image_a = rng.normal(size=(n_img, d))
image_b = rng.normal(size=(n_img, d))
text    = rng.normal(size=(n_txt, d))
text2   = rng.normal(size=(n_txt, d))

out_a = causal_attention(np.concatenate([image_a, text]))
out_b = causal_attention(np.concatenate([image_b, text]))
out_t = causal_attention(np.concatenate([image_a, text2]))

print("change the image -> text outputs change:", not np.allclose(out_a[n_img:], out_b[n_img:]))
print("change the text  -> image outputs change:", not np.allclose(out_a[:n_img], out_t[:n_img]))

change the image -> text outputs change: True
change the text  -> image outputs change: False

Toy model. Random vectors and a single attention layer. Real models stack many layers, and some allow image tokens to attend to each other in both directions; the point here is only what a causal mask implies.

So in this layout the instruction is read in the light of the image, which is what a VLA needs.

Fusion by modulation: FiLM

There is another way to let words steer vision. Feature-wise Linear Modulation (FiLM) influences a network’s computation through a simple feature-wise affine transformation based on conditioning information.^[5] In plain terms: the instruction produces a scale and a shift for each feature of the image, and the features are multiplied by the scale and the shift is added. RT-1 used a FiLM-conditioned EfficientNet,^[6] which is how the same photo can be processed differently depending on the instruction.

import numpy as np
rng = np.random.default_rng(0)

features = rng.normal(size=(5, 6))                      # 5 image positions, 6 features each
instruction = {"pick the red block": np.array([1.0, 0, 0]),
               "open the drawer":    np.array([0, 1.0, 0])}

W_scale = rng.normal(size=(3, 6))                       # maps the instruction to a scale per feature
W_shift = rng.normal(size=(3, 6))                       # ... and to a shift per feature

outputs = {}
for text, vec in instruction.items():
    gamma, beta = 1 + vec @ W_scale, vec @ W_shift      # scale and shift
    outputs[text] = gamma * features + beta
    print(text, "-> first feature of position 0:", round(float(outputs[text][0, 0]), 3))

a, b = outputs.values()
print("same image, different instruction, different features:", not np.allclose(a, b))

pick the red block -> first feature of position 0: 1.8
open the drawer -> first feature of position 0: -1.245
same image, different instruction, different features: True

Toy model. The weights are random, so only the mechanism is real: one image, two instructions, two different feature sets.

Concatenating tokens and FiLM both let language influence vision. Which is used depends on the model; the next lesson looks at how the result becomes an action.

Common misconceptions

Engineering takeaways

1. Embedding = a row of numbers for a token.
2. Concatenate image and text vectors and attention does the fusion.
3. A causal mask lets text read the image but not the reverse when the image comes first.
4. FiLM scales and shifts image features using the instruction.

Fusion means that words and pixels can finally affect each other.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: how the fused representation decides what the robot does.

Back to the VLA roadmap