M6.3 — Robot Demonstrations and Trajectory Datasets

Learning objectives

After this lesson, you should be able to:

• Define demonstration, episode, trajectory and dataset.
• Name common ways demonstrations are collected.
• Estimate the size of a dataset from episodes, steps and image size.
• Align sensor streams with different rates by nearest timestamp.
• Build action chunks and normalise actions with training-set statistics.
• Explain why validation data should be split by episode.

The one-sentence idea

A demonstration dataset is a set of episodes whose time-aligned observations and actions become the examples a policy learns from, and how it is built and split decides how far the results can be trusted.

The dataset defines what the policy can do.

What a demonstration is

A demonstration is a recording of someone performing the task with the robot. It is stored as an episode: a sequence of time steps from the start of the attempt to its end. At every step the recording holds what the robot observed and the action that was taken, as M5.1 described, usually together with a language instruction for the episode and a note of whether the attempt succeeded. A trajectory is the path through states and actions that the episode traces out, and a dataset is a collection of such episodes, from tens to many thousands.

Demonstrations are produced in several ways:

Teleoperation

A person drives the robot remotely with a joystick, a 3-D mouse, a VR controller or a handheld device. The most common source of large datasets.

Leader-follower arms

A person moves a lightweight “leader” arm and the robot’s “follower” arm copies it. Gives natural, fine motions.

Kinesthetic teaching

A person physically guides the robot’s arm by hand while it is recorded. Simple, but the person is in the way of the cameras.

Scripted or planned

A program generates the motion, often in simulation. Cheap and plentiful, but only as varied as the script.

The shape of the data

An episode is a set of arrays that share a leading time axis. A typical one stores camera frames, the robot’s joint readings, the actions, and the instruction. Because the camera frames dominate the size, it is worth doing the arithmetic:

import numpy as np

T_ = 300                                                    # 300 steps, e.g. 30 seconds at 10 Hz
episode = {
    "image":       np.zeros((T_, 3, 224, 224), dtype=np.uint8),    # a camera frame per step
    "joint_pos":   np.zeros((T_, 7), dtype=np.float32),            # proprioception
    "action":      np.zeros((T_, 8), dtype=np.float32),            # 7 joint targets and a gripper command
    "instruction": "put the red cup in the bin",
    "success":     True,
}
print({k: v.shape for k, v in episode.items() if hasattr(v, "shape")})

episodes = 100
frames = episodes * T_
print(frames)                                               # training examples
print(round(frames * 3 * 224 * 224 / 1e9, 2), "GB of raw pixels")

{'image': (300, 3, 224, 224), 'joint_pos': (300, 7), 'action': (300, 8)}
30000
4.52 GB of raw pixels

One hundred modest episodes already give 30,000 (observation, action) examples, and the pixels alone would fill about 4.5 GB uncompressed. In practice images are stored compressed, often as video, and decoded as the data is loaded. Common storage layouts include HDF5 files, TFRecord-based formats such as RLDS (the format used by the Open X-Embodiment collection that Phase 5 covers), and Parquet-plus-video formats such as the one used by the LeRobot library. What they share is the same episode-and-step structure above.

Putting sensors on one clock

Each sensor runs at its own rate: a camera at 30 Hz, joint encoders at 100 Hz or more, a policy at 10 Hz. Before training, the streams have to be aligned on a common timeline, usually by taking, for each step, the reading nearest in time. If the alignment is off by even a few frames, the image and the action describe slightly different moments, and the policy learns a relationship that is not real:

cam_t = np.arange(0, 1.0, 1 / 30)                           # camera timestamps, 30 Hz
joint_t = np.arange(0, 1.0, 1 / 100)                        # joint timestamps, 100 Hz
joint_val = np.sin(2 * np.pi * joint_t)                     # a joint signal

nearest = np.abs(joint_t[None, :] - cam_t[:, None]).argmin(axis=1)    # index of the closest joint reading
aligned = joint_val[nearest]
print(len(cam_t), aligned.shape)
print(np.abs(joint_t[nearest] - cam_t).max().round(4))     # worst timing mismatch, in seconds

30 (30,)
0.0033

Each of the 30 camera frames is paired with the closest joint reading, at most 3.3 ms away (the worst case for a 100 Hz stream is 5 ms). Timestamps need to be recorded at capture, and clocks across devices need to agree, or the mismatch can be tens of milliseconds, which is large compared with how fast a robot arm moves.

From episodes to training examples

Behavior cloning (M6.2) needs (observation, action) pairs, and each time step provides one. Models that predict a short sequence of future actions at once, called action chunks (Phase 4), instead take a window of the next H actions as the target. Near the end of an episode there are fewer than H actions left, so the window is padded by repeating the last one:

def make_chunks(actions, H):
    T_ = len(actions)
    padded = np.concatenate([actions, np.repeat(actions[-1:], H - 1, axis=0)])   # repeat the last action
    return np.stack([padded[t:t + H] for t in range(T_)])                       # (T, H, action_dim)

acts = np.arange(5, dtype=float).reshape(5, 1)             # five one-number actions: 0, 1, 2, 3, 4
chunks = make_chunks(acts, 3)
print(chunks.shape)
print(chunks[:, :, 0])

(5, 3, 1)
[[0. 1. 2.]
 [1. 2. 3.]
 [2. 3. 4.]
 [3. 4. 4.]
 [4. 4. 4.]]

Row t is the target chunk for time step t. The last two rows repeat the final action, since the episode ends. A policy that predicts such chunks outputs (H, action_dim) numbers per observation, and the shape (T, H, action_dim) is what a dataset of chunked actions looks like.

Actions also need normalising, as in M2.2, because joint velocities, positions and gripper values have different scales. The statistics are computed once over the whole training set and applied everywhere, including at deployment:

rng = np.random.default_rng(0)
acts = np.column_stack([rng.normal(0.2, 0.05, 1000),       # one joint: small, centred away from zero
                        rng.normal(0.0, 2.0, 1000)])       # another: much larger scale
mean, std = acts.mean(axis=0), acts.std(axis=0)
norm = (acts - mean) / std
print(mean.round(2), std.round(2))
print(np.allclose(norm.mean(axis=0), 0), norm.std(axis=0).round(2))

[ 0.2  -0.02] [0.05 2.05]
True [1. 1.]

Before normalising, one joint’s values varied about 40 times more than the other’s, so a loss on the raw values would be dominated by the large-scale joint. After normalising, each has mean 0 and standard deviation 1. The saved mean and std become part of the model: at run time a predicted normalised action is multiplied by std and shifted by mean to give a real command (the decoding of Phase 4).

Quality, splits and what to watch for

The data a policy learns from caps what it can do, so a few checks are routine:

• Idle time. Demonstrations often start and end with the operator doing nothing. A policy trained on long idle stretches learns to do nothing. Such steps can be found by the size of the action and trimmed.
• Consistency. If operators solve the task in different ways, the policy faces the multimodal problem of M6.2.
• Coverage. The objects, positions, lighting and instructions in the dataset set the range in which the policy can be trusted.
• Failures. Episodes marked unsuccessful are usually dropped, or used separately.

acts = np.concatenate([np.zeros((20, 2)), rng.normal(0, 0.3, (60, 2)), np.zeros((20, 2))])   # idle, motion, idle
moving = np.linalg.norm(acts, axis=1) > 1e-3
first, last = moving.argmax(), len(moving) - moving[::-1].argmax()
print(len(acts), first, last, last - first)

100 20 80 60

Of 100 steps, 20 at the start and 20 at the end are idle, leaving 60 steps of real motion to keep. The last, and most important, practice is how the data is split for evaluation (M6.4). Consecutive steps in an episode are nearly identical, so splitting by step puts near-copies of training examples in the validation set. Split by episode instead:

rng = np.random.default_rng(1)
walks = [np.cumsum(rng.normal(0, 0.05, (100, 2)), axis=0) + rng.normal(0, 1, 2) for _ in range(10)]   # 10 episodes
ep = np.repeat(np.arange(10), 100)
X = np.concatenate(walks)

def median_nearest_train_distance(val_mask):
    tr, va = X[~val_mask], X[val_mask]
    d = np.linalg.norm(va[:, None, :] - tr[None, :, :], axis=2).min(axis=1)
    return round(float(np.median(d)), 3)

by_step = rng.random(len(X)) < 0.2                          # random 20% of steps for validation
by_episode = ep >= 8                                        # the last 2 episodes for validation
print(median_nearest_train_distance(by_step), median_nearest_train_distance(by_episode))

0.021 0.439

With a step-wise split, a typical validation point has a training point within 0.021 of it, so the model has effectively seen it. With an episode-wise split the nearest training point is 0.439 away, a fair test of whether the model works on a trajectory it has not seen. A step-wise split would give an optimistic score that says little about how the robot will do on a new attempt.

Common misconceptions

Engineering takeaways

1. An episode is a time-aligned sequence of observations and actions, with an instruction and a success flag.
2. Demonstrations come from teleoperation, leader-follower arms, kinesthetic teaching or scripts.
3. Images dominate dataset size, so they are usually stored compressed.
4. Streams at different rates are aligned by nearest timestamp, and bad alignment teaches false relationships.
5. Action chunks are windows of future actions, padded at the end of an episode.
6. Normalise actions with training-set statistics, and reuse them to decode predictions.
7. Trim idle steps and check consistency and coverage.
8. Split validation data by episode, not by step.

The dataset is the specification the policy actually learns, so inspect it as carefully as the model.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You know what a demonstration dataset contains. Next is how to train on it and measure the result honestly with training and validation data.

Back to the VLA roadmap prerequisites