10.5 — Learned Policies and Whole-Body Control

Learning objectives

After this lesson, you should be able to:

• Explain why the VLA is separate from whole-body control.
• Describe learned controllers trained in simulation and transferred to hardware.
• Simulate balance with and without feedback.
• Explain command modes and history-based adaptation.

The one-sentence idea

The VLA chooses; the whole-body controller keeps the robot standing while it does.

Balance cannot wait for a slow model.

The VLA does not hold the robot up

A tabletop arm stays where it is put. A humanoid does not: it balances on two feet and, if the control stops for a moment, it falls. So humanoid systems separate what to do (a VLA choosing reaching and grasping targets at a few to tens of Hz) from how to keep the whole body consistent (a whole-body controller running much faster, tracking the targets while keeping balance and respecting contacts). Figure's Helix, for example, is described with a fast System 1 producing continuous actions at 200 Hz.^[1] The whole-body controller can be classical optimisation-based control, or learned. This lesson is about the learned kind and how it fits under a VLA.

Learned whole-body and locomotion controllers

Two research results show the pattern. Hwangbo et al. trained a neural-network controller in simulation by reinforcement learning and transferred it to the ANYmal quadruped, where it followed commanded velocities more precisely, ran faster than earlier methods and recovered from falls. They note that RL had mostly stayed in simulation because training on hardware is costly and slow.^[2] For humanoids, Radosavovic et al. built a fully learned walking controller: a transformer that reads the robot's recent proprioceptive observations and actions and predicts the next action, trained with large-scale model-free reinforcement learning in randomised simulated environments and deployed on the real Digit robot without further tuning. It handled various outdoor terrains and external disturbances.^[3]

HOVER takes a different angle on the interface. Humanoid control needs different command modes (root velocity for navigation, upper-body joint angles for tabletop work); instead of one policy per mode, it uses full-body kinematic motion imitation as a shared abstraction and distils the modes into a single policy, reported to outperform specialised controllers.^[4]

Why balance needs a fast loop

A standing humanoid behaves like an inverted pendulum: lean a little and gravity makes you lean more. The simplest model is θ″ = (g/ℓ)·θ + torque + disturbance. Without control, any lean grows exponentially. A reaching arm shifts the centre of mass, which is a disturbance on the body. A fast PD balance controller counters it:

import numpy as np

g, length, dt = 9.81, 1.0, 0.002

def simulate(kp, kd, disturbance, seconds=3.0):
    theta, omega = 0.0, 0.0
    for k in range(int(seconds / dt)):
        t = k * dt
        push = disturbance if 0.5 <= t < 0.7 else 0.0                      # the arm swings out for 0.2 s
        torque = -kp * theta - kd * omega                                  # the balance controller
        alpha = (g / length) * theta + torque + push
        omega += alpha * dt; theta += omega * dt
        if abs(theta) > 0.5: return "FELL at t = %.2f s" % t
    return "upright, final lean %.4f rad" % theta

print("no balance controller:     ", simulate(0, 0, 1.0))
print("balance controller (PD):   ", simulate(40, 8, 1.0))

no balance controller:      FELL at t = 1.47 s
balance controller (PD):    upright, final lean 0.0000 rad

Toy model. A linearised single inverted pendulum. A real humanoid has many joints and contact forces; this shows only why a fast feedback loop is needed.

With no control the robot has fallen by 1.47 s, about a second after the arm swings out; with a PD loop it recovers and ends upright. This is the job of the lower layer, and it needs to run at a high, fixed rate regardless of how slowly the policy thinks.

Command spaces: what the policy asks the body for

The VLA must say what it wants in a form the whole-body controller accepts. HOVER's point is that there are several such forms and they overlap. A small illustration of selecting which parts of a target are tracked in each mode:

parts = ["root velocity", "upper-body joints", "lower-body joints", "hand positions"]
modes = {
    "navigation":         {"root velocity"},
    "tabletop":           {"upper-body joints", "hand positions"},
    "loco-manipulation":  {"root velocity", "hand positions"},
    "full-body imitation": set(parts),
}
print(f"{'mode':<20}" + "".join(f"{p:<20}" for p in parts))
for name, tracked in modes.items():
    print(f"{name:<20}" + "".join(f"{('tracked' if p in tracked else '-'):<20}" for p in parts))

mode                root velocity       upper-body joints   lower-body joints   hand positions      
navigation          tracked             -                   -                   -                   
tabletop            -                   tracked             -                   tracked             
loco-manipulation   tracked             -                   -                   tracked             
full-body imitation tracked             tracked             tracked             tracked             

Toy model. A hand-made table of modes for illustration, loosely following the examples in the HOVER abstract; the paper defines its own modes.

The mask idea is the same as in RDT-1B's action layout (lesson 6.5): one interface, with a mask for what each command carries.

Learning from history instead of knowing the physics

Radosavovic's controller reads recent proprioceptive history, and the idea behind that is that history carries information about the environment (a slippery floor, a heavy load) that the network can use to adapt in context.^[3] A toy: a hidden extra mass makes a joint respond more slowly than expected. A controller that estimates the response from the last few steps compensates; one that assumes nominal mass does not:

import numpy as np
rng = np.random.default_rng(0)

def run(estimate_from_history, mass, steps=60):
    pos, vel, target = 0.0, 0.0, 1.0
    history = []
    gain = 1.0                                                          # the controller's belief about 1 / mass
    for t in range(steps):
        cmd = 20.0 * (target - pos) - 6.0 * vel
        if estimate_from_history and len(history) >= 5:
            accels = np.array([a for a, c in history[-5:]]); cmds = np.array([c for a, c in history[-5:]])
            gain = float(np.clip((cmds @ accels) / (cmds @ cmds + 1e-9), 0.2, 2.0))   # least-squares estimate of 1 / mass
        u = cmd / gain                                                  # compensate for the believed response
        accel = u / mass
        history.append((accel, u))
        vel += accel * 0.02; pos += vel * 0.02
    return abs(target - pos)

for mass in (1.0, 3.0):
    print(f"hidden mass {mass}: distance from the target after 1.2 s -> assumes nominal mass {run(False, mass):.3f} | estimates from history {run(True, mass):.3f}")

hidden mass 1.0: distance from the target after 1.2 s -> assumes nominal mass 0.029 | estimates from history 0.029
hidden mass 3.0: distance from the target after 1.2 s -> assumes nominal mass 0.254 | estimates from history 0.035

With nominal mass both do the same; with a heavy load the history-based version adapts. A transformer reading a long history can learn this kind of in-context adaptation without being told the physics.

Common misconceptions

Engineering takeaways

1. VLA above, whole-body controller below.
2. Simulation-trained controllers transferred to hardware.
3. Inverted pendulum needs feedback.
4. History enables adaptation.

Choose slowly; balance quickly.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: balance, contact constraints and embodiment-specific actions.

Back to the VLA roadmap