11.3 — Cross-Embodiment Transfer

Learning objectives

After this lesson, you should be able to:

• State the evidence for cross-embodiment transfer and its limits.
• Compute workspace overlap between two arms.
• Resample motions between control rates.
• Explain why per-robot statistics are required.

The one-sentence idea

Share what is common; handle what differs.

Bodies differ in reach, rate, numbers and cameras.

One policy, many bodies

Training a separate policy for every robot wastes data. The hope behind cross-embodiment learning is that robots share a lot of task structure, such as where objects are and what a good grasp is, even when their bodies differ, so a policy trained on many robots could transfer to a new one. Evidence exists in specific settings: the Open X-Embodiment authors report that a high-capacity model trained on pooled data (22 embodiments) shows positive transfer, improving several robots using experience from other platforms;^[1] Octo is reported to fine-tune to new sensory inputs and action spaces within a few hours on consumer GPUs;^[2] and Gemini Robotics is reported to adapt to new robot embodiments with further fine-tuning.^[3] These are results for particular robots and tasks, not a promise for yours.

What stands in the way? Bodies differ in at least four ways that a model must handle: how far and where they can reach, how fast they are controlled, how their numbers are defined and what their cameras see.

Difference 1: what each robot can reach

A demonstration recorded on a large arm may ask for a hand position that a small arm cannot reach at all. Transfer is limited to the overlap of what the two bodies can do. For a planar two-link arm, the reachable region is a ring between |L1 − L2| and L1 + L2 from the shoulder. How much of a large arm's demonstrated targets can a smaller arm reach?

import numpy as np
rng = np.random.default_rng(0)

def reachable(points, L1, L2):
    r = np.linalg.norm(points, axis=1)
    return (r >= abs(L1 - L2)) & (r <= L1 + L2)

# targets demonstrated on a large arm (links 0.6 m and 0.5 m), placed over its workspace
angles = rng.uniform(0, 2 * np.pi, 5000)
radii = rng.uniform(0.1, 1.1, 5000)
targets = np.column_stack([radii * np.cos(angles), radii * np.sin(angles)])
targets = targets[reachable(targets, 0.6, 0.5)]

for name, (L1, L2) in (("same arm", (0.6, 0.5)), ("smaller arm (0.4 + 0.35)", (0.4, 0.35)), ("much smaller arm (0.25 + 0.2)", (0.25, 0.2))):
    print(f"{name:<30} can reach {100 * reachable(targets, L1, L2).mean():5.1f}% of the demonstrated targets")

same arm                       can reach 100.0% of the demonstrated targets
smaller arm (0.4 + 0.35)       can reach  64.5% of the demonstrated targets
much smaller arm (0.25 + 0.2)  can reach  34.7% of the demonstrated targets

Toy model. A planar two-link arm and uniformly spread targets. Real workspaces are 3D and have joint limits, but the overlap argument is the same.

A policy cannot learn from a demonstration that its body cannot perform. Good pipelines filter such cases or map targets into the smaller robot's workspace.

Difference 2: how fast each robot is controlled

One robot is controlled at 5 Hz, another at 50 Hz. The same physical motion is 5 numbers per second for one and 50 for the other. A chunk of 10 actions covers two seconds on the first and 0.2 seconds on the second. Data must be resampled to a common rate (lesson 7.2), and a model must know the rate. A smooth motion resampled by interpolation:

import numpy as np

t5 = np.arange(0, 1.0001, 1 / 5)                       # robot A: 5 Hz
motion_a = np.sin(np.pi * t5)                          # the same physical motion, sampled at 5 Hz
t50 = np.arange(0, 1.0001, 1 / 50)                     # robot B: 50 Hz
resampled = np.interp(t50, t5, motion_a)               # A's motion on B's clock

truth = np.sin(np.pi * t50)
print("numbers per second: robot A", len(t5) - 1, "| robot B", len(t50) - 1)
print("a 10-action chunk covers:", 10 / 5, "s on robot A and", 10 / 50, "s on robot B")
print("largest error from straight-line resampling:", round(float(np.abs(resampled - truth).max()), 3))

numbers per second: robot A 5 | robot B 50
a 10-action chunk covers: 2.0 s on robot A and 0.2 s on robot B
largest error from straight-line resampling: 0.049

Straight-line resampling loses some curvature: fine motions at a slow rate cannot be recovered at a fast rate. Tell the model the rate, or make sure the data cover similar rates.

Difference 3: how the numbers are defined

This is lessons 4.1 and 4.3 again: joint or hand space, absolute or delta, units and normalisation. RDT-1B's answer is a physically interpretable unified action space that represents different robots' actions in one format while preserving the physical meaning of each original action,^[4] so slots line up across robots. Even then each robot keeps its own statistics:

import numpy as np
rng = np.random.default_rng(0)

robots = {"robot A": (0.02, 0.005), "robot B": (0.002, 0.0006)}        # typical step size (m) and spread, invented
for name, (mean, std) in robots.items():
    steps = np.abs(rng.normal(mean, std, 1000))
    print(f"{name}: typical step {steps.mean() * 1000:5.2f} mm, 99th percentile {np.quantile(steps, 0.99) * 1000:5.2f} mm")

print("a normalised action of 1.0 means a different distance on each robot, so decoding needs the right statistics")

robot A: typical step 19.76 mm, 99th percentile 31.14 mm
robot B: typical step  2.00 mm, 99th percentile  3.35 mm
a normalised action of 1.0 means a different distance on each robot, so decoding needs the right statistics

A normalised action of 1.0 is 20 mm on one robot and 2 mm on the other. Mixing them without per-robot statistics, as in lesson 5.3, would produce commands ten times too big or too small.

Difference 4: what the cameras see

Camera placement changes the image of the same scene (lesson 9.3). A policy trained on many camera placements can learn to cope, which is why diverse datasets help; one trained on a single placement ties its sense of space to that placement. A move of 1 degree already shifts the image by about 10 pixels at typical focal lengths (lesson 10.4). If the new robot's camera differs a lot, expect to recollect some data or fine-tune. What to do in practice, with the evidence above: start from a multi-robot pretrained model, adapt with the new robot's own demonstrations (lesson 7.4), check the action definition and statistics (lessons 4.1, 4.3), and evaluate with the generalisation tests of phase 8.

Common misconceptions

Engineering takeaways

1. Positive transfer reported in specific settings.
2. Overlap of reach limits transfer.
3. Resample to a common rate.
4. Keep per-robot statistics.

Share the task, translate the body.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: world models, which predict what will happen so a robot can plan before it acts.

Back to the VLA roadmap