7.2 — Dataset Preparation and Preprocessing

Learning objectives

After this lesson, you should be able to:

• Write rules that flag bad episodes.
• Resample sensors to a common clock.
• Compute and save statistics from the training split.
• Make a reproducible split by episode.

The one-sentence idea

Clean, align, measure, split, and write it all down.

Every step must be recorded to be reproducible.

Data preparation is where most of the work is

A policy is only as good as its demonstrations (lesson 3.2). Raw recordings are messy: cameras and joint sensors run at different rates, some episodes are failures, some have frozen frames, units differ. Standard formats exist to hold clean data: RLDS is a toolkit for recording, replaying, editing and sharing data for sequential decision making, built around a standardised, lossless format,^[1] and LeRobot’s dataset format stores synchronised camera video and tabular state and action data.^[2] This lesson walks through four preparation steps in code: clean, align, compute statistics, split.

Step 1: find the bad episodes

Look for episodes that are too short, contain long pauses, or have frozen sensors. Simple rules catch most of them:

import numpy as np
rng = np.random.default_rng(0)

def make_episode(length, frozen_after=None, gap_at=None):
    t = np.arange(length) * 0.1
    if gap_at is not None: t[gap_at:] += 2.0                       # a 2 second hole in the timestamps
    joints = np.cumsum(rng.normal(0, 0.01, (length, 3)), axis=0)
    if frozen_after is not None: joints[frozen_after:] = joints[frozen_after]    # sensor stopped updating
    return {"t": t, "joints": joints}

episodes = {"ok": make_episode(200), "too short": make_episode(12), "frozen sensor": make_episode(200, frozen_after=80), "time gap": make_episode(200, gap_at=100)}

def problems(ep, min_len=50, max_gap=0.5):
    out = []
    if len(ep["t"]) < min_len: out.append("too short")
    if np.diff(ep["t"]).max() > max_gap: out.append("timestamp gap")
    frozen = np.all(np.diff(ep["joints"], axis=0) == 0, axis=1)
    if frozen.sum() > 0.2 * len(frozen): out.append("sensor frozen for more than 20% of the episode")
    return out

for name, ep in episodes.items():
    print(f"{name:<14} ->", problems(ep) or "looks fine")

ok             -> looks fine
too short      -> ['too short']
frozen sensor  -> ['sensor frozen for more than 20% of the episode']
time gap       -> ['timestamp gap']

Do not delete silently: log what was dropped and why, and keep the list. A dataset whose cleaning rules are unrecorded cannot be reproduced.

Step 2: put everything on one clock

The camera may run at 30 Hz and the joint sensor at 100 Hz. Training needs one observation-action pair per step, so resample to a common rate by interpolating onto shared timestamps:

import numpy as np

camera_t = np.arange(0, 1.0, 1/30)                                  # 30 Hz frames
joint_t  = np.arange(0, 1.0, 1/100)                                 # 100 Hz joint readings
joint_pos = np.sin(2 * np.pi * joint_t)                             # one joint

aligned = np.interp(camera_t, joint_t, joint_pos)                   # joint position at each camera time
print("camera frames:", len(camera_t), "| joint readings before:", len(joint_t), "| aligned joint values:", len(aligned))
print("largest difference from the true value at those times:", round(float(np.abs(aligned - np.sin(2*np.pi*camera_t)).max()), 4))

camera frames: 30 | joint readings before: 100 | aligned joint values: 30
largest difference from the true value at those times: 0.0004

Interpolation is fine for smooth joint signals, but not for things like gripper open/close commands (use the latest value instead), and never for images (pick the nearest frame).

Step 3: compute statistics from the training split only

Normalisation statistics (lesson 4.3) are computed from data, so compute them from the training episodes only, otherwise information about the test set leaks into the model, and save them next to the model:

import json
import numpy as np
rng = np.random.default_rng(0)

actions_by_episode = [rng.normal(0.0, 0.02, (100, 3)) for _ in range(10)]
train, test = actions_by_episode[:8], actions_by_episode[8:]            # split by episode

stacked = np.concatenate(train)
stats = {"mean": stacked.mean(axis=0).round(5).tolist(),
         "std": stacked.std(axis=0).round(5).tolist(),
         "q01": np.quantile(stacked, 0.01, axis=0).round(5).tolist(),
         "q99": np.quantile(stacked, 0.99, axis=0).round(5).tolist()}
text = json.dumps(stats)                                                 # this string is what you would write to a file
restored = json.loads(text)
print("statistics keys:", sorted(restored))
print("round trip exact:", restored == stats)
print("computed from", len(train), "training episodes and", len(stacked), "steps")

statistics keys: ['mean', 'q01', 'q99', 'std']
round trip exact: True
computed from 8 training episodes and 800 steps

The file with these numbers must travel with the checkpoint. Version both together.

Step 4: split by episode, once, and write it down

Lesson 3.2 showed why the split must be by episode. A fixed random seed makes the split reproducible:

import numpy as np

episode_ids = np.arange(50)
rng = np.random.default_rng(1234)                                        # a recorded seed
shuffled = rng.permutation(episode_ids)
test, val, train = shuffled[:5], shuffled[5:10], shuffled[10:]

print("train/val/test episodes:", len(train), len(val), len(test))
print("no overlap:", len(set(train) & set(val)) == 0 and len(set(train) & set(test)) == 0 and len(set(val) & set(test)) == 0)
print("test episodes:", sorted(test.tolist()))

train/val/test episodes: 40 5 5
no overlap: True
test episodes: [10, 32, 34, 38, 48]

Write the test list into the dataset's metadata and never train on it. Then you can say, truthfully, what your numbers mean.

A checklist before you train

Before the first training run, go through a short list. Is the action definition written down (joint or hand space, absolute or delta, units, lesson 4.1)? Does every episode carry its instruction text? Are camera names and image sizes the same across episodes? Is there a list of dropped episodes and the reason for each? Are the statistics computed from training episodes only and stored with the dataset? Is the split saved, with the seed? And has someone looked at a few episodes as video, not just as numbers? A frame-by-frame look at three or four episodes catches problems no rule will: a camera pointing the wrong way, a lighting change halfway, an operator who gave up and walked away.

None of this is glamorous, but it decides whether the later lessons on fine-tuning and evaluation tell you anything true. If the data are wrong, a lower loss only means the model learned the wrongness better.

Common misconceptions

Engineering takeaways

1. Flag, do not silently delete.
2. One clock.
3. Statistics from training episodes only.
4. Seeded episode split.

Prepare the data so that someone else can redo it.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: running a pretrained policy.

Back to the VLA roadmap