1.3 — LLM Planner vs. VLM vs. VLA

Learning objectives

After this lesson, you should be able to:

• State the inputs and outputs of an LLM, a VLM and a VLA.
• Explain why a language planner needs a feasibility signal.
• Choose the next skill by combining usefulness and feasibility.
• Explain why a VLA’s abilities depend on its robot training data.

The one-sentence idea

The three model types differ by what they write: text, text about images, or robot actions.

Only a VLA writes the action itself.

Three kinds of model, three interfaces

The three names describe what a model takes in and what it gives back. A large language model (LLM) reads and writes text. A vision-language model (VLM) reads images and text and writes text; PaliGemma, built from a SigLIP image encoder and the Gemma-2B language model, is an example.^[1] A vision-language-action model (VLA) reads images and text and writes robot actions, as defined in lesson 1.2.^[2] Only the last one has an output that a robot can execute directly.

interfaces = {
    "LLM": ("text",                 "text"),
    "VLM": ("image + text",         "text"),
    "VLA": ("image + text",         "robot action"),
}
print(f"{'model':<6}{'reads':<16}{'writes'}")
for name, (reads, writes) in interfaces.items():
    print(f"{name:<6}{reads:<16}{writes}")

model reads           writes
LLM   text            text
VLM   image + text    text
VLA   image + text    robot action

The difference in the last column is the important one. Text from an LLM or VLM has to be turned into something a motor can use, by a person, by other software, or by training the model to emit actions itself.

The LLM as a planner

The problem. A language model knows a lot about tasks in general. Ask it how to clean up a spill and it will say: get a sponge, wipe the spill, throw the sponge away. What it does not know is the robot’s situation right now. It cannot tell that the sponge is on the far side of the room, out of the arm’s reach. A plan that sounds sensible can be impossible to carry out from where the robot is standing.

The fix (the idea behind SayCan). Ask two separate questions about every skill the robot already has, and combine the answers:

• Is it useful? Would this skill help with the goal? A language model answers this, because it knows about tasks.
• Is it feasible? Could the robot do it successfully from its current position? A learned “can I do this?” estimate for each skill answers this, because it knows about the robot.^[3]

SayCan multiplies the two numbers for each skill and picks the skill with the highest product. The chosen skill is added to the plan, the situation is updated, and the process repeats until the best choice is a “done” step.^[3]

Why multiply? Multiplying works like “and”: a skill only scores well if it is useful and feasible. If either number is close to zero, the product is close to zero, so a very useful skill that cannot be done right now loses to a modest one that can.

A picture of what the code below does. The goal is “clean up the table”. The robot starts far from the table. Picking up the sponge is the most useful skill, but it is out of reach, so the robot first chooses to walk to the table. Once it is there, the sponge is reachable and it picks it up. The plan emerges one step at a time.

The skills and all the numbers are made up for illustration. A real system gets “useful” from a language model and “feasible” from learned value functions.

# The robot's "menu": the skills it already knows how to do.
skills = ["pick up the sponge", "pick up the cup", "go to the table", "done"]

# Goal given to the robot: "clean up the table".

# USEFUL: how much would this skill help with the goal? 0 = not at all, 1 = exactly what is needed.
# (In a real system a language model supplies this number. Here we just make it up.)
useful = {
    "pick up the sponge": 0.60,   # the sponge is what we need to wipe the table
    "go to the table":    0.30,   # helpful, but only as a way to get close to things
    "pick up the cup":    0.05,   # not part of the goal
    "done":               0.05,   # the table is not clean yet
}

# FEASIBLE: how likely is this skill to work from where the robot stands right now? 0 = no chance, 1 = certain.
# (In a real system a learned model supplies this. Here we make it up too.)
feasible = {
    "pick up the sponge": 0.10,   # the sponge is on the table, far away: the arm cannot reach it
    "go to the table":    0.95,   # walking across the room is easy
    "pick up the cup":    0.10,   # also out of reach
    "done":               0.90,   # the robot could stop at any time
}

def show(title):
    # Print one row per skill and pick the skill with the highest score.
    print(title)
    print(f"{'skill':<20}{'useful':>8}{'feasible':>10}{'score':>8}")
    best_skill, best_score = None, -1.0
    for skill in skills:
        score = useful[skill] * feasible[skill]      # the rule: score = useful x feasible
        print(f"{skill:<20}{useful[skill]:>8.2f}{feasible[skill]:>10.2f}{score:>8.3f}")
        if score > best_score:                       # remember the best one so far
            best_skill, best_score = skill, score
    print("robot chooses:", best_skill)
    print()

# What would we pick if we looked at usefulness alone?
print("usefulness alone would pick:", max(useful, key=useful.get), "(but the robot cannot reach it)")
print()

# Step 1: the robot is far from the table.
show("Step 1: robot is far from the table")

# Step 2: the robot has walked to the table, so the sponge is now within reach.
# Only this one number changes.
feasible["pick up the sponge"] = 0.85
show("Step 2: robot is now at the table")

usefulness alone would pick: pick up the sponge (but the robot cannot reach it)

Step 1: robot is far from the table
skill                 useful  feasible   score
pick up the sponge      0.60      0.10   0.060
pick up the cup         0.05      0.10   0.005
go to the table         0.30      0.95   0.285
done                    0.05      0.90   0.045
robot chooses: go to the table

Step 2: robot is now at the table
skill                 useful  feasible   score
pick up the sponge      0.60      0.85   0.510
pick up the cup         0.05      0.10   0.005
go to the table         0.30      0.95   0.285
done                    0.05      0.90   0.045
robot chooses: pick up the sponge

Toy model. The skills and every number are invented. This reproduces the idea of multiplying usefulness by feasibility from SayCan, not its implementation or results.

How to read the output. The first line shows what would happen without the feasibility check: the robot would try to pick up a sponge it cannot reach. In step 1 the sponge scores only 0.60 × 0.10 = 0.06, while “go to the table” scores 0.30 × 0.95 = 0.285, so the robot walks. In step 2 only one number changed (the sponge is now reachable, 0.10 becomes 0.85), and the sponge scores 0.60 × 0.85 = 0.51, which wins. In this toy the “go to the table” number stays at 0.95 for simplicity; a real system would recompute every number after each step.

What this shows. A language planner is “grounded” by checking its ideas against what the robot can actually do. Language models can also write code for this layer: Code as Policies shows that code-writing language models can turn natural-language commands into robot policy code that calls perception and control APIs.^[4] And the planner can be told what happened: Inner Monologue adds closed-loop language feedback from sources such as success detectors, object detectors and humans, which significantly improves instruction completion and lets the planner replan around failures.^[5]

A VLM that talks about the robot’s world

PaLM-E feeds images, robot state estimates and text, interleaved, into a pretrained language model trained end to end on tasks that include robot manipulation planning, visual question answering and captioning. Its outputs are text.^[6] So it sees what the robot sees and can answer or plan in words, but a separate controller still has to carry the words out. In the taxonomy of this lesson it is a VLM with robot inputs, not a VLA.

The VLA closes the gap

RT-2 trains a vision-language model to output the robot’s actions as text tokens, so that language and action are handled in the same way.^[2] Surveys of the area organise VLAs by components and research lines, with the term credited to RT-2.^[7] The planner/VLM split stays useful in practice: a system may use a language model for high-level steps and a VLA to carry out each step. How one robot system divides the work is a design choice, not part of the definitions, and the architecture lesson 1.4 returns to it.

Because a VLA is trained to output actions, what it can do depends heavily on the robot data it was trained on, which is why the data discussion in lessons 1.1 and 1.2 matters. The RT-2 paper reports that the model’s physical skills remain limited to the distribution of skills seen in the robot data.^[2]

Common misconceptions

Engineering takeaways

1. LLM: text to text. VLM: image and text to text. VLA: image and text to action.
2. A planner needs both usefulness and feasibility.
3. PaLM-E is a VLM-style model with robot inputs.
4. What a VLA can do depends on its robot training data.

Ask what the model writes, and you know which kind it is.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next, the components inside a VLA and how information flows from the camera to the motor.

Back to the VLA roadmap