10.4 — VLA Policies for Humanoid Manipulation

Learning objectives

After this lesson, you should be able to:

• Estimate the action size and token count for a humanoid.
• Explain why a head camera needs head pose in the observation.
• Explain why bimanual errors compound.
• Name the data challenges for humanoids.

The one-sentence idea

Same VLA ideas, more joints and more ways to be inconsistent.

Include head and body pose; budget for two-arm error.

What changes on a humanoid

Everything in phases 1 to 9 applies to a humanoid, but four things get harder: more degrees of freedom (two arms, two hands, a torso, a neck, and legs), two arms that must cooperate, a camera on a moving head, and a body that can fall over. GR00T N1 was deployed on the Fourier GR-1 humanoid for language-conditioned bimanual manipulation tasks,^[1] Helix's System 1 is stated to output continuous robot actions at 200 Hz,^[2] and RDT-1B targets bimanual manipulation with an action layout that preserves each joint's physical meaning.^[3] This lesson looks at the manipulation side; balance is the subject of lessons 10.5 and 10.6.

More degrees of freedom, more numbers

Degrees of freedom depend on the configuration. For the Unitree G1, listings give a base configuration of 23 DoF (two 6-DoF legs, two 5-DoF arms and a 1-DoF waist), with 43 DoF when fitted with dexterous hands, and other variants in between; the count for a given unit must be checked against its configuration.^[4] What a larger action vector costs a model:

configs = {"two arms only (2 x 7)": 14, "G1 base, 23 DoF": 23, "G1 with hands, 43 DoF": 43}
chunk_steps = 30
for name, dims in configs.items():
    tokens = dims * chunk_steps
    print(f"{name:<24} {dims:>2} numbers per step -> {tokens:>4} numbers (= tokens, one per number) for a {chunk_steps}-step chunk")

two arms only (2 x 7)    14 numbers per step ->  420 numbers (= tokens, one per number) for a 30-step chunk
G1 base, 23 DoF          23 numbers per step ->  690 numbers (= tokens, one per number) for a 30-step chunk
G1 with hands, 43 DoF    43 numbers per step -> 1290 numbers (= tokens, one per number) for a 30-step chunk

Toy model. The 14 is an assumed two-arm layout, and not every DoF of a humanoid is commanded by the policy: lower-body joints are often handled by a separate controller (lesson 10.5).

Hands add the most. Hand motion is also smooth and high-frequency, which is the case where per-number binning performs poorly according to the FAST paper (lesson 4.8),^[5] one reason continuous or compressed outputs are common on humanoids.

A camera that moves with the head

A fixed table camera sees the same object at the same pixel every time. A head camera does not: turning the head by 1 degree shifts the image by roughly f·tan(1°) pixels, and the same object appears elsewhere. If the policy does not know the head angle, it cannot tell “the object moved” from “the head moved”:

import numpy as np

f = 600.0                                                       # focal length in pixels
for degrees in (1, 5, 15):
    shift = f * np.tan(np.radians(degrees))
    print(f"head turned {degrees:>2} deg -> an object straight ahead moves by about {shift:5.1f} pixels in the image")

head turned  1 deg -> an object straight ahead moves by about  10.5 pixels in the image
head turned  5 deg -> an object straight ahead moves by about  52.5 pixels in the image
head turned 15 deg -> an object straight ahead moves by about 160.8 pixels in the image

Two remedies: give the policy the head (and torso) pose as part of the state, so it can account for it, or control the head with a separate behaviour that keeps the workspace in view. Either way, the observation definition must say where the camera is and how it can move (lesson 9.3).

Two arms must agree

In a handover or a two-handed lift, both hands must hit the same place. If each arm independently has a position error of about 1 cm, the gap between the hands is larger than either error alone, because the errors add (in variance):

import numpy as np
rng = np.random.default_rng(0)

error_per_arm = 0.01                                           # 1 cm standard deviation per axis, per arm
left = rng.normal(0, error_per_arm, (100000, 3))
right = rng.normal(0, error_per_arm, (100000, 3))

gap = np.linalg.norm(left - right, axis=1)
print("typical error of one arm (RMS distance):", round(float(np.sqrt(np.mean(np.sum(left**2, axis=1)))) * 100, 2), "cm")
print("typical gap between the two hands (RMS):", round(float(np.sqrt(np.mean(gap**2))) * 100, 2), "cm")
print("ratio:", round(float(np.sqrt(np.mean(gap**2)) / np.sqrt(np.mean(np.sum(left**2, axis=1)))), 2), "(about the square root of 2)")

typical error of one arm (RMS distance): 1.73 cm
typical gap between the two hands (RMS): 2.46 cm
ratio: 1.42 (about the square root of 2)

The gap is about 1.4 times a single arm's error. Tasks where the hands must meet or jointly hold something are less forgiving than single-arm tasks, which is one reason bimanual data is scarce and models target it specifically; RDT-1B was fine-tuned on a self-created bimanual dataset of over 6K episodes.^[3]

Humanoid data

Collecting humanoid demonstrations means teleoperating many joints at once, often with a motion-capture or VR rig, and the human's body does not match the robot's. That is one reason heterogeneous sources like human videos and synthetic data are used alongside robot data.^[1] It also means that the action definition (lesson 4.1) and the retargeting from human to robot motion must be documented with the data.

Common misconceptions

Engineering takeaways

1. 43 DoF with hands on G1.
2. Hands need continuous or compressed outputs.
3. Give head pose to the policy.
4. Two-arm gaps are ~1.4× single errors.

More body, more things to keep consistent.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: how learned policies combine with whole-body control.

Back to the VLA roadmap