P9-T3 — Open-Vocabulary Detection and Vision-Language Models

Learning objectives

After this lesson, you should be able to:

• Explain how an image-text embedding model scores an image against text prompts.
• Compute cosine similarity and the softmax probabilities that follow, with a temperature.
• Describe open-vocabulary detection and how it differs from a fixed-class detector.
• List the failure modes of language-grounded perception and the checks a robot needs.

The idea: images and words in one space

Two encoders, one for images and one for text, are trained on hundreds of millions of image-caption pairs so that a picture and its caption land close together in a shared vector space, and unrelated pairs land far apart. The idea behind CLIP and its successors is that the closeness is measured by the angle between the vectors, the cosine similarity. See the VLA track’s lesson on CLIP for the training objective; here we use the trained result.

Zero-shot classification

To classify an image into any set of classes, write a prompt for each (“a photo of a traffic cone”), embed them, and pick the prompt whose vector is closest to the image’s. The classes are text, so a new class is a new sentence, not a new training run.

Open-vocabulary detection and grounding

An open-vocabulary detector localises objects named by a text query. It combines an image encoder that proposes regions or tokens with a text encoder for the query, and scores each region against each query. The result is a set of boxes with scores for whatever you asked for. Grounding models go further and tie phrases of a sentence (“the red box on the left shelf”) to regions. A robot can then be told what to look for in words, which is the perception half of a language-driven robot.

How they fail

They fail in ways a fixed-class detector does not.

1. Prompt sensitivity. “A cone”, “a traffic cone” and “an orange cone” can give different scores, and wording that was common in the training captions works best.
2. Relations and counting. They are good at what is in the picture and poor at relations (left of, behind), counts and negation (“a crate with no lid”).
3. Over-confidence. The softmax over prompts always gives a winner, with high probability, even when none of the prompts is correct, because the probabilities are relative to the list.
4. Look-alikes and rare things. Similar objects, fine distinctions (which mug) and objects rare on the web are confused.
5. Biases and gaps inherited from web data, including poor coverage of industrial and robot-eye views.

What a robot does with the output

The output is a box and a score from a model with no calibrated notion of “none of these”. It enters the pipeline like any detection: it must be lifted to 3D, aligned, filtered, checked against other sensors, and tracked, as in the capstone. A command such as “pick up the blue mug” should be confirmed by checking that the detection is stable over time, that the object has the right size and position, and, where an error would matter, asking the person (the grounding and asking lessons of the Physical AI track).

The mathematics: cosine similarity, temperature and softmax

Let u be the image embedding and t₁ … t_K the text embeddings of the K prompts. The similarity of the image with prompt k is the cosine of the angle between the vectors:

u · t_k

the dot product of the image and text vectors.

‖u‖, ‖t_k‖

their lengths; dividing by them makes s_k depend only on the angle, between −1 and 1.

s_k

the similarity: 1 for the same direction, 0 for perpendicular vectors.

The probabilities over the prompts come from a softmax of the similarities, scaled by a temperature (written as a multiplier τ, often about 100 in trained models):

τ

the scale (inverse temperature): a large τ makes the distribution sharp, so small similarity differences become large probability differences.

p_k

the probability assigned to prompt k, relative to the other prompts in the list only.

The probabilities add to 1 over the given prompts. They say nothing about whether any prompt describes the image. To allow “none of these” one adds a background prompt, or thresholds the raw similarity s_k (compared with a value found on validation data), not the softmax.

Detection. For region proposals r with embeddings u_r, the score of region r for query k is s_rk as above; boxes with the score above a threshold are kept, then non-maximum suppression is applied, as in Lesson P6-T2.

Worked examples: a similarity that looks decisive and is not

A softmax from small differences. An image of a traffic cone has cosine similarities 0.30, 0.25 and 0.20 with the prompts “a traffic cone”, “a crate” and “a person”. With τ = 100 the logits are 30, 25 and 20. Subtracting the largest, the weights are e⁰ = 1, e⁻⁵ = 0.0067 and e⁻¹⁰ = 0.000045, so

A difference of 0.05 in similarity, a small number, gives a 99.3% probability to the first prompt. The model looks certain. Is it? Take the same image and the prompts “a traffic cone”, “a crate” and “a person” when the real object is a barrel: the similarities might be 0.22, 0.20 and 0.12; the logits are 22, 20 and 12, and the probabilities are 0.88, 0.12 and 0.00004. The model says “traffic cone, 88%” about a barrel, because the barrel is not in the list.

The remedy: thresholds on the raw similarity. Validation data on your robot show that correct matches have s ≥ 0.26 and wrong ones typically fall below 0.23. Rejecting detections with s < 0.25 removes the barrel as a “cone” (0.22) and keeps the true cone (0.30). The softmax alone would have accepted both. The threshold must come from validation data in your conditions, as in Lab P6-L3.

Prompt sensitivity. The same cone scores 0.30 for “a traffic cone” and 0.24 for “a cone”, and 0.27 for “an orange cone”. With a fixed threshold of 0.25 the second wording fails. Practical systems evaluate several prompts per class and average them, and fix the wording on validation data, not by trial on one image.

A robot command. “Pick up the blue mug.” The open-vocabulary detector returns three boxes: the blue mug (score 0.31), a blue bowl (0.28) and a red mug (0.27). The scores are close. A robust system does not just take the top box: it checks the box against other cues (colour in HSV space, Lesson P2-T1; size; position on the table), tracks it for several frames, and asks which one if two remain plausible. A confident grasp of the bowl would be a perception failure that a good language model downstream cannot fix.

Engineering implications and limitations

Validate on your own data. Collect images from your robot of the objects you will name, with ground truth, and measure precision and recall of the open-vocabulary model per class and condition, as in Module 6. Do not rely on public benchmark results.

Choose thresholds on the raw similarity. Use the validation set to set a threshold per class, or a margin between the best and second-best prompts. Include look-alike and unrelated objects as negatives.

Include a background and negatives. Add prompts for what the robot will often see but should not act on (“a wall”, “a shelf”, “the floor”), so that the softmax has somewhere to put it.

Fix and test the prompt wording. Prompt templates (“a photo of a {class}”) matter. Test variants on validation data and fix them with the software.

Feed the output into the normal pipeline. Lift, align, track, fuse and gate (Modules 3, 7 and 8). A text-prompted box is a detection like any other and has the same failure modes plus the ones above.

Ask when unsure. If the top two candidates are close, or the best score is under the threshold, let the system ask or take a second look instead of acting (Physical AI track).

Keep the scope clear. No vision-language model is run in the labs. They teach the pipeline that such a model’s output would enter.

Common misconceptions

Engineering takeaways

1. Image and text encoders share an embedding space; the cosine similarity s = u·t/(‖u‖‖t‖) scores an image against a prompt.
2. Softmax with a scale τ ≈ 100 turns a 0.05 similarity gap into 99.3% vs 0.7%; the probabilities are relative to the prompt list only.
3. Open-vocabulary detectors score regions against text queries and extend a detector beyond a fixed class list; classes are sentences.
4. They fail on wording, relations, counts, look-alikes and rare things, and are over-confident: threshold the raw similarity, include negatives, and validate on your data.
5. Their output enters the normal pipeline (lifting, alignment, tracking, fusion) and needs confirmation before it drives an action.

References and further reading

Papers for the models named. The similarities, thresholds and detections in the examples are constructed to show the arithmetic; they are not outputs of any model.

• Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML (CLIP).
• Minderer, M. et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV (OWL-ViT).
• Liu, S. et al. (2024). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV.
• Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what the perception system must deliver for manipulation and navigation, and how the requirements differ.

Related content

Nothing below is required to finish this lesson.

• Before
  P9-T2: Vision Transformers and Foundation Models

  Describe attention, pretraining and what foundation models change.