M4.6 — Multimodal Representations and Visual Grounding

Learning objectives

After this lesson, you should be able to:

• Describe the shared-space, one-sequence and cross-attention ways of combining modalities.
• Explain the role of a projector between a vision encoder and a language model.
• Read a prefix attention mask for image and text tokens.
• Explain visual grounding and compute a grounded pixel from a patch similarity map.
• Back-project a pixel with depth to a 3-D point using the pinhole model.
• Name the main grounding failure modes.

The one-sentence idea

A multimodal model turns images and text into tokens of the same kind, reads them as one sequence, and can then tie a phrase in the instruction to a specific place in the scene.

Same kind of token, one sequence, and the phrase finds its place.

Modalities and how to combine them

A modality is a kind of input: text, images, depth, audio, robot joint readings. A VLA is multimodal because it takes camera images and a text instruction, often also the robot’s own state, and produces actions. The central design question is how to combine the modalities into something a single network can reason over. Three patterns cover most models:

A shared space (late fusion)

Each modality has its own encoder, and a training objective pulls their outputs into one space, as in CLIP (M4.5). Good for matching and retrieval, but each side is summarised separately before they meet.

One sequence (early fusion)

Convert every modality into tokens and place them in one sequence that a single transformer reads. Every token can attend to every other, so words and image patches interact in every layer.

Cross-attention

One sequence supplies the queries, and the other supplies the keys and values (M3.3). The language side reads from the image without the two being concatenated.

Many recent vision-language models, and the VLAs built on them, use the one-sequence pattern with a pretrained language model as the backbone. The rest of this lesson follows that design.

Feeding image features to a language model

A vision encoder produces feature vectors of its own width (for example 768), while a language model expects embeddings of its width (for example 4096). A small projector, a linear layer or a short MLP, maps the first to the second. After that, image tokens and text tokens are the same kind of object and can simply be concatenated:

import torch
from torch import nn

torch.manual_seed(0)
vision_dim, lm_dim = 24, 16                       # tiny stand-ins for e.g. 768 and 4096
image_feats = torch.randn(1, 196, vision_dim)     # a vision encoder's output: 196 patch features
text_embeds = torch.randn(1, 12, lm_dim)          # embeddings of 12 instruction tokens

projector = nn.Linear(vision_dim, lm_dim)
image_tokens = projector(image_feats)
sequence = torch.cat([image_tokens, text_embeds], dim=1)
print(image_tokens.shape, sequence.shape)
print(768 * 4096 + 4096)                          # parameters of a 768 -> 4096 linear projector

torch.Size([1, 196, 16]) torch.Size([1, 208, 16])
3149824

The 196 image tokens and 12 text tokens form a single sequence of 208 tokens, which the language model processes as usual. At real sizes, a 768 → 4096 projector holds about 3.1 million parameters, small next to the billions in the backbone. During training the projector is often learned first, with the encoder and language model frozen, and the rest is fine-tuned afterwards (M3.5).

One detail concerns the attention mask (M3.3). The image tokens do not have a “past” and “future”, so they can attend to one another freely. The text tokens should still be generated left to right. A common arrangement lets everything attend to the image prefix, and keeps the causal rule for the text that follows:

import numpy as np

n_img, n_txt = 2, 3
T_ = n_img + n_txt
allowed = np.tril(np.ones((T_, T_), dtype=int))      # causal: each token sees itself and earlier tokens
allowed[:n_img, :n_img] = 1                          # image tokens see all the image tokens
print(allowed)

[[1 1 0 0 0]
 [1 1 0 0 0]
 [1 1 1 0 0]
 [1 1 1 1 0]
 [1 1 1 1 1]]

Rows are the tokens doing the looking and columns the tokens being looked at. The two image tokens see each other and nothing of the text. Each text token sees both image tokens and the text before it, never the text after it. That is how a model can describe, or act on, a scene it can see in full.

Visual grounding: linking words to what is visible

Visual grounding means connecting a phrase in the instruction to the specific thing it refers to in the image. “The red cup” is not a concept to be recognised in general. It points at one particular object, in one particular place in the current scene, and a robot has to reach exactly there. The same question appears in the Physical AI track as grounding and when to ask. Here the focus is on how the visual features make it possible.

A simple way to see grounding in a patch-based encoder (M4.3) is to compare the phrase’s embedding with the feature of every patch. The patch that matches best is where the phrase most likely refers to. The vectors below are hand-made to show the mechanism for a 2 × 3 grid of patches:

patch_feats = np.array([[[0.1, 0.9, 0.0], [0.0, 0.1, 0.9], [0.1, 0.8, 0.1]],     # row 0: three patches
                        [[0.1, 0.9, 0.1], [0.9, 0.1, 0.0], [0.0, 0.1, 0.9]]])    # row 1
phrase = np.array([1.0, 0.1, 0.0])                                               # embedding of "the red cup"

def unit_last(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

sim = unit_last(patch_feats) @ unit_last(phrase)       # (2, 3) similarity map
row, col = (int(i) for i in np.unravel_index(sim.argmax(), sim.shape))
print(sim.round(2))
print(row, col)

patch = 16                                              # pixels per patch side
print((col * patch + patch / 2, row * patch + patch / 2))   # centre of that patch as (u, v) pixels

[[0.21 0.01 0.22]
 [0.21 1.   0.01]]
1 1
(24.0, 24.0)

The similarity map peaks at row 1, column 1, so the phrase is grounded in that patch, whose centre is pixel (u, v) = (24, 24). This is the idea, not a complete system: real models do the equivalent inside their attention layers, and may output a box or a mask instead of a patch.

For a robot, a pixel is not yet an action. Turning it into a point in 3-D space uses the depth at that pixel and the camera’s intrinsic parameters, with the pinhole model: a pixel (u, v) at depth Z lies at X = (u − cx) Z / fx and Y = (v − cy) Z / fy, where f is the focal length in pixels and (cx, cy) the image centre. Phase 9 of the roadmap covers coordinate frames and calibration properly. This is the shape of the calculation:

u, v, Z = 400, 300, 1.5                  # pixel position and depth in metres
fx = fy = 600.0                          # focal length in pixels
cx, cy = 320.0, 240.0                    # image centre
X = (u - cx) * Z / fx
Y = (v - cy) * Z / fy
print(round(X, 3), round(Y, 3), Z)

0.2 0.15 1.5

The point is 0.2 m to the right, 0.15 m below the camera’s optical axis and 1.5 m ahead, expressed in the camera’s frame. A real robot then converts it into the robot’s own frame before moving there.

Where grounding goes wrong

• Ambiguity. “The cup” when two cups are visible has no single answer. A careful system asks, or uses context, instead of guessing.
• Attribute binding. Knowing that “red”, “blue”, “cup” and “bowl” are all present is not the same as knowing that it is the cup that is red. Models that summarise a sentence as a collection of word features can lose this, as the toy example below shows.
• Spatial relations. “Left of”, “behind” and “on top of” depend on layout, which global features discard (M4.3) and contrastive training captures weakly (M4.5).
• Hallucination. A language-model backbone can describe objects that are not in the image, because it is trained to produce fluent text. A robot that acts on such a description reaches for nothing.

word = {"red": [1, 0, 0, 0], "blue": [0, 1, 0, 0], "cup": [0, 0, 1, 0], "bowl": [0, 0, 0, 1]}

def bag(sentence):
    return np.sum([word[w] for w in sentence.split()], axis=0)

a = bag("red cup blue bowl")
b = bag("blue cup red bowl")
print(a, b, np.array_equal(a, b))

[1 1 1 1] [1 1 1 1] True

Two different scenes, a red cup with a blue bowl and a blue cup with a red bowl, have identical bag-of-words vectors. The sequence-based models of this roadmap keep word order and position, and attention lets “red” connect to “cup”, which is why they can do better. It remains a known weak spot, so evaluation (Phase 8) checks perception, grounding and action failures separately.

Common misconceptions

Engineering takeaways

1. Multimodal models combine modalities by a shared space, one token sequence, or cross-attention.
2. A projector maps vision features to the language model’s embedding width so that image and text tokens can be concatenated.
3. A prefix mask lets image tokens attend freely while text stays causal.
4. Visual grounding ties a phrase to a specific place in the image, for example the patch with the highest similarity.
5. The pinhole model turns a pixel and its depth into a 3-D point in the camera frame.
6. Ambiguity, attribute binding, spatial relations and hallucination are the main failure modes.
7. A bag of words cannot bind attributes, and sequence models with attention can do better.
8. A grounded target still needs frames and calibration before the robot can move.

To act on a sentence, the robot has to find the thing it refers to, not just recognise the words.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the vision and language prerequisites. The next module covers robotics fundamentals: state, frames, kinematics and control.

Back to the VLA roadmap prerequisites