VLA · Phase 3 · Robot Learning Foundations
3.4 — Training Objectives, Loss Functions and Evaluation
What the model is optimised for, and how to tell whether it actually works.
A loss tells the model what to improve; it does not tell you if the robot works. This lesson covers the common losses, a trap with disagreeing demonstrations, and how to read a success rate honestly.
Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.
Learning objectives
After this lesson, you should be able to:
- Explain cross-entropy on action tokens and squared error on continuous actions.
- Explain why squared error averages disagreeing demonstrations.
- Explain why low loss does not imply task success.
- Interpret a success rate from a small number of trials.
- Predict an action for a demonstrated state
- Compare it with the demonstrator’s action (the loss)
- Update the model to reduce the loss
- Run the model on its own
- Measure the task success rate, with uncertainty
The one-sentence idea
The loss trains the model; only task success tells you if it works.
Report success rates, and say how many trials were behind them.
What “training” optimises
Training means choosing model parameters that make a number small: the loss. For behavior cloning (lesson 3.3) the loss compares the model’s predicted action with the demonstrator’s. What you compare, and how, depends on how the action is represented. If the action is a list of real numbers, the usual loss is the average squared difference (mean squared error, MSE) or the average absolute difference (L1). If the action is a token, as in OpenVLA, which is trained with next-token prediction on action tokens,[1] the loss is cross-entropy, the same as for language: the model gives a probability to every possible token and is penalised by minus the log of the probability it gave to the right one.
import numpy as np
# The model's probabilities over 5 action bins; the demonstrator's action was bin 3
probs_confident = np.array([0.01, 0.02, 0.05, 0.90, 0.02])
probs_unsure = np.array([0.20, 0.20, 0.20, 0.20, 0.20])
probs_wrong = np.array([0.80, 0.10, 0.05, 0.03, 0.02])
for name, p in (("confident and right", probs_confident), ("completely unsure", probs_unsure), ("confident and wrong", probs_wrong)):
print(f"{name:<20} loss = {-np.log(p[3]):.2f}")Output:
confident and right loss = 0.11 completely unsure loss = 1.61 confident and wrong loss = 3.51
Being right and confident costs almost nothing; being confidently wrong costs a lot. That is the behaviour you want from a training signal.
A trap: when demonstrations disagree
Suppose a person steers round an obstacle sometimes on the left (action −1) and sometimes on the right (+1), equally often. Both are fine. What does a model trained with squared error do? It predicts the average, which is 0, straight into the obstacle:
import numpy as np
demos = np.array([-1.0] * 50 + [1.0] * 50) # half go left, half go right
candidates = np.linspace(-1, 1, 5)
for a in candidates:
mse = np.mean((demos - a) ** 2)
print("predict", a, "-> squared error", round(float(mse), 2))
print("lowest squared error at:", candidates[np.argmin([np.mean((demos - a) ** 2) for a in candidates])])Output:
predict -1.0 -> squared error 2.0 predict -0.5 -> squared error 1.25 predict 0.0 -> squared error 1.0 predict 0.5 -> squared error 1.25 predict 1.0 -> squared error 2.0 lowest squared error at: 0.0
Toy model. Two exact modes and a single obstacle. Real data is messier but the issue is real: a loss that rewards the average can produce an action that no demonstrator ever took.
This is one reason to prefer outputs that are a distribution rather than a single number. Binned tokens, as in lesson 1.2, give each option its own probability, so “left 50%, right 50%” can be represented. Some policies instead generate actions: π0 generates action chunks with flow matching,[2] which is a different kind of output from a single regressed number. Later lessons return to these action heads.
Low loss is not success
The loss measures how well the model matches demonstrations at the demonstrated states. Success is a different question: does the robot complete the task when it runs on its own, visiting its own states? In lesson 3.3 the cloned driver matched the expert exactly on the recorded states, but left the road in about 3 runs in 10. Measure both:
import numpy as np
rng = np.random.default_rng(0)
def expert(e): return -1.8 * e
def drive(policy, noise, steps=40):
e, seen = 0.0, []
for _ in range(steps):
seen.append(e)
e = 1.5 * e + policy(e) + rng.normal(0, noise)
if abs(e) > 1.0:
return seen, True
return seen, False
states = np.concatenate([drive(expert, 0.03)[0] for _ in range(30)])
actions = expert(states)
cloned = lambda e: actions[np.argmin(np.abs(states - e))]
test_states = np.concatenate([drive(expert, 0.03)[0] for _ in range(10)]) # fresh expert demonstrations
loss = np.mean([(cloned(e) - expert(e)) ** 2 for e in test_states])
success = 1 - np.mean([drive(cloned, 0.12)[1] for _ in range(200)])
print("held-out loss on expert states:", round(float(loss), 5))
print("success rate when driving alone:", round(float(success), 3))Output:
held-out loss on expert states: 0.0 success rate when driving alone: 0.695
A loss that rounds to zero and a success rate of about 70%. The loss is a training signal; the success rate is the number that matters. That is why robot papers report task success. OpenVLA, for example, is compared with RT-2-X by the absolute task success over 29 tasks and several robot embodiments.[1]
How many trials do you need?
Success rates from real robots come from a limited number of trials, so they carry uncertainty. If a robot succeeds in 7 of 10 tries, the true rate could easily be 50% or 90%. A standard way to quantify this is the Wilson confidence interval for a proportion. The interval below shows how it narrows with more trials, for the same observed rate of 70%:
import numpy as np
def wilson(successes, n, z=1.96): # 95% interval for a success rate
p = successes / n
centre = (p + z*z/(2*n)) / (1 + z*z/n)
half = z * np.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1 + z*z/n)
return centre - half, centre + half
for n in (10, 50, 200):
lo, hi = wilson(round(0.7 * n), n)
print(n, "trials, 70% success -> true rate between", round(float(lo), 2), "and", round(float(hi), 2))Output:
10 trials, 70% success -> true rate between 0.4 and 0.89 50 trials, 70% success -> true rate between 0.56 and 0.81 200 trials, 70% success -> true rate between 0.63 and 0.76
With 10 trials, a policy that scored 70% could be anywhere from about 40% to 89%. Comparing two policies that differ by a few points needs many more trials than robot labs can usually afford, so be sceptical of small differences from few trials.
Common misconceptions
Misconception 1 “Lower loss always means a better robot.”
Correction In the experiment the loss was almost zero and the robot still failed in about 3 runs in 10.
Misconception 2 “Mean squared error is always the right loss.”
Correction With disagreeing demonstrations it predicts their average, which may be an action nobody took.
Misconception 3 “70% success is 70% success.”
Correction From 10 trials the plausible range is about 40% to 89%.
Engineering takeaways
- Tokens use cross-entropy; real numbers use MSE or L1.
- Squared error averages modes.
- Loss is measured on demonstrated states; success on the robot’s own.
- Few trials give wide uncertainty.
Train on loss, judge on success.
Knowledge check
Try each question first, then reveal the explanation.
Question 1
A model gives the right action token probability 0.5. What is the cross-entropy loss, to two decimals?
Explanation
0.69. −ln 0.5 ≈ 0.693.
Question 2
Half of demonstrations go left (−1), half go right (+1). What action minimises squared error?
Explanation
0. The average, which goes in neither demonstrated direction.
Question 3
Why can a policy with near-zero loss still fail?
Explanation
The loss only covers states in the demonstrations; the policy’s own mistakes lead it to other states.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2024). OpenVLA: An open-source vision-language-action model. Conference on Robot Learning (CoRL) 2024. arXiv:2406.09246
Cited for:
- Each action dimension is split into 256 bins between the 1st and 99th quantile of the training actions; the 256 bins are written over the 256 least-used tokens of the Llama tokenizer, and the model is trained with next-token prediction on the action tokens. Checked against the paper, via summaries of its text.
- Outperforms the 55B-parameter RT-2-X by 16.5 percentage points of absolute task success across 29 tasks and several robot embodiments, with 7 times fewer parameters. Checked against the paper’s abstract.
(2024). π0: A vision-language-action flow model for general robot control. arXiv:2410.24164
Cited for:
- Pairs the pretrained vision-language model with a separate set of weights for the action outputs (the “action expert”) that generates action chunks with flow matching. Checked against the paper, via summaries of its text.
What’s next
Last in this phase: distribution shift, why a policy that works on its training data can fail in a slightly different setting.