M2-T6 — Splits, Leakage and Robot Data

Learning objectives

After this lesson, you should be able to:

• Explain why a random step-level split leaks on episode-structured and time-ordered robot data, and quantify the leak.
• Split by episode, day, robot or floor, and use group k-fold when data are scarce.
• Name four other leaks: normalisation statistics, labels computed from the future, duplicates and tuning on the test set.
• Write a data-card checklist that records how a dataset was collected and split.

The intuition

The word “independent” hides the whole problem. A split estimates how the model will do on new data only if the validation examples are drawn the way the future examples will be. The future examples will be new episodes: new starts, new people in the hall, a different day. They will not be new steps of episodes the model has already seen.

So the unit that must be kept whole is the episode (or the day, or the robot, whichever the future will change). Cut between units, never inside one.

The mathematics

Why neighbouring steps are near-duplicates

Within an episode the robot moves a short distance each step, so consecutive observations nearly coincide and consecutive actions nearly coincide. A flexible model can answer a validation step by recalling the neighbouring training steps. This is interpolation, which is easy, while the future requires extrapolation to new episodes, which is hard. Episode-level persistent factors (a demonstrator’s style, the day’s lighting, a calibration offset) make it worse, because every step of the episode carries the same fingerprint.

Kinds of split

Group k-fold

When there are few groups (episodes, days, robots), split them into k folds. For each fold, train on the other k − 1 groups and score on the held-out group; every example is scored once. The mean of the k scores is the estimate and their spread is its uncertainty:

For time-ordered data use forward chaining instead: train on days 1 to j, score on day j + 1, and slide the window. Never train on a day after the one you score.

Other ways to leak

• Normalisation: computing the mean and standard deviation over all the data, then splitting, pushes information about the validation set into the training inputs. Compute statistics on the training set only and apply them unchanged to validation and test (lesson M2-T2).
• Labels computed from the future: a label such as “will collide within 2 s” or “return from this step” is built from later steps. If the features include anything computed with later steps (a smoothed velocity centred on the step, a filter run forwards and backwards), the feature already contains the label. At deployment those later steps do not exist yet.
• Duplicates: the same log copied into two folders, overlapping windows, or an episode re-exported at two frame rates will fall on both sides of a random split. Deduplicate before splitting, or split by source file.
• Tuning on the test set: choosing the model, the threshold or the stopping point by looking at the test score turns it into a validation set (lesson M2-T5).
• Selection: evaluating only on episodes that completed, or on starts the expert found easy, flatters the model (lesson M1-T6).

Validation error is not task success

A lower mean squared error on actions does not guarantee a higher closed-loop success rate, because small errors compound as the policy visits states the data never contained (module M3). Whatever the split, the final evaluation of a policy is a rollout on held-out scenarios with an interval (lesson M1-T5).

A worked example

Numbers below were computed with RLCore in Node. Dataset: 12 expert episodes from starts(21, 12, 'all'), 446 steps in all (episodes of 23 to 72 steps). To make the data like real teleoperated demonstrations, each episode’s actions carry a persistent operator offset drawn once per episode (standard deviation 0.25 in each component) plus independent noise of 0.05 per step. The expected irreducible action error is therefore about 0.25² + 0.05² = 0.065 for any model that cannot know which episode it is in. Four models predict the action from the four-number observation: 1-nearest-neighbour and 15-nearest-neighbour on scaled observations, and two linear policies, one on pillar-aware features and one on goal-direction features (lesson M1-T4).

1. The data are near-duplicates. For 446 steps, the nearest other step (scaled observation distance) lies in the same episode for 143, 32%, although a randomly chosen other step would be in the same episode only 9% of the time (the other 11 episodes hold 91% of the steps). The median distance to the nearest neighbour is 0.015 in scaled units. Consecutive actions differ by 0.112 on average; two random steps differ by 0.521.

2. Two splits, same models (mean squared action error, mean and standard deviation over 30 random splits; step-level holds out a random quarter of the steps, episode-level holds out 3 of 12 episodes):

Three things stand out. First, the step-level errors for three of the four models, 0.039 to 0.046, are below the 0.065 floor that no model can beat on new episodes: the models are partly recalling each episode’s offset from its neighbours. A validation error below the known noise floor is a red flag. Second, the episode-level errors are about 1.7 to 2.0 times larger for the three flexible models, and 1.2 times for the least flexible one (the direction-feature linear model, 0.067 to 0.083): the leak grows with capacity. Third, the step-level numbers are tight (a standard deviation of 0.006 or 0.007) and wrong; the episode-level standard deviations (0.026 to 0.053) are as large as the gaps between models, so with 12 episodes the episode-level validation cannot separate the models either.

3. Closed loop. The last column trains each model on all 12 episodes and rolls it out from 100 starts that were not used (starts(900, 100, 'all')). Success ranges from 46 to 100 of 100, and crashes from 0 to 54. Action error does not rank these models the way success does: the 15-nearest-neighbour model has the lowest validation error under both splits and only 75% success, while the pillar-aware linear model, with a similar validation error, succeeds in all 100. A mean squared action error averages over steps, while a crash is decided at the few steps next to the pillar (lesson M1-T6).

4. Group k-fold. Four folds of three episodes each. For 1-nearest-neighbour the fold errors are 0.148, 0.065, 0.118 and 0.072, mean 0.100, standard deviation 0.039. For the pillar-aware linear model they are 0.141, 0.032, 0.153 and 0.036, mean 0.091, standard deviation 0.065. Folds 1 and 3 are the hard ones for both models; this lesson did not investigate why. Which three episodes you hold out changes the answer by a factor of 2.3 for the nearest-neighbour model and 4.8 for the linear one, which is the honest message of a dataset this small: the uncertainty is large, and a single split would hide it.

5. Normalisation leak. A synthetic battery-voltage log: ten days, 50 readings a day, voltage 12.6 − 0.12 × day plus noise of 0.05 V. Train on days 1 to 8, validate on days 9 and 10. Standardising with training statistics (mean 12.064, standard deviation 0.284) puts the validation mean 2.13 standard deviations below the training mean, which is what a model would meet in deployment. Standardising with statistics from all ten days (mean 11.943, standard deviation 0.352) shrinks the apparent shift to 1.37 standard deviations. The pooled statistics have quietly folded the future into the inputs and made the drift look smaller. In the 12-episode robot data the same mistake changed the mean of the goal offset dx from 3.316 to 3.372, almost nothing, because those episodes are not ordered by drift: whether a leak matters depends on the data, and you cannot tell without cutting correctly.

What the simulation simplifies. The operator offset is a deliberately simple model of a persistent per-episode effect. The hall has one obstacle and noise-free dynamics. A 12-episode dataset is small, and the numbers show it: every episode-level estimate has a wide spread. The direction of the effects (a low step-level error, a higher episode-level error, success not tracking either) is robust across the settings tried; the exact figures are for this seed.

Engineering implications and limitations

Decide the unit of the future first

Ask what will differ between training and deployment: new episodes, a later date, another robot, another floor. Split along that axis, and keep every group intact. If you cannot name it, split by episode as a minimum.

Keep a held-out set that is hard on purpose

Reserve some episodes from the most different conditions: the other floor, the evening shift, the second robot. They tell you about transfer, which the random episodes do not.

Fit every preprocessing step on training data only

Normalisation, feature selection, outlier thresholds, even the choice of which sensors to drop: put them inside the cross-validation loop. A pipeline that normalises first and splits second has leaked.

Audit labels and features for the future

For each feature, ask “would this value exist at the moment the policy acts?”. Filters run offline with non-causal smoothing, labels derived from episode outcomes and “time to goal” fields are the usual culprits.

Deduplicate by source

Hash episodes, or split by source file. Check that no recording appears on both sides, including resampled or re-exported copies.

Write a data card

1. What was recorded: robot, sensors, rates, units, coordinate frames, software version.
2. How: autonomous, teleoperated or scripted; who operated; on which days, floors and robots.
3. How many: episodes, steps, per day, per condition, with counts of failed and discarded episodes and why.
4. Splits: the rule used (by episode, day, robot), the random seed, the fold assignment and the date the test set was frozen.
5. Preprocessing: what was fitted and on which data; filters, smoothing, labelling, and whether any use later steps.
6. Known gaps: starts, speeds or conditions the data do not cover (module M8 returns to this).

A data card is cheap to write when the data are collected and impossible to reconstruct a year later. Lab M2-L3 asks for the checks that find the leak and the overfit in a log that scored 99 percent.

Common misconceptions

Engineering takeaways

1. Neighbouring steps of one episode are near-duplicates (the nearest step was in the same episode 32% of the time); split by episode, day, robot or site, never inside one.
2. On the Scout demonstrations the random step-level validation error (0.046) was about half the episode-level one (0.088), and below the 0.065 noise floor.
3. Use group k-fold when data are few, forward chaining for time, and report the spread across folds; 12 episodes gave fold errors from 0.065 to 0.148.
4. Fit normalisation and every other preprocessing step on training data only, avoid features and labels built from later steps, deduplicate, and touch the test set once.
5. Validation error is not success: always finish with a closed-loop rollout on held-out scenarios with an interval, and keep a data card.

References and further reading

Sources on data leakage, splits for structured data, and dataset documentation.

• S. Kaufman, S. Rosset, C. Perlich and O. Stitelman, “Leakage in Data Mining: Formulation, Detection, and Avoidance”, ACM Transactions on Knowledge Discovery from Data 6(4), 2012.
• D. R. Roberts et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure”, Ecography 40(8), 2017.
• S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science”, Patterns 4(9), 2023.
• T. Gebru et al., “Datasheets for Datasets”, Communications of the ACM 64(12), 2021.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: Module M3 uses these skills to learn a policy from demonstrations, and shows what happens when the cloned policy leaves the states in the data. Put them into practice in labs M2-L2 and M2-L3 first.

Related content

Nothing below is required to finish this lesson.

• Before
  M2-T5: Generalization and Overfitting

  Tell a model that fits from one that generalizes, using train, validation and test errors.

• Lab
  M2-L3: Catch the Overfit Model

  The model scores 99 percent on logged data and fails on the floor: find the leak and the overfit, and fix the evaluation.