M1-T2 — Observations, Actions and Datasets

Learning objectives

After this lesson, you should be able to:

• Distinguish an observation from the true state, and say when the difference matters.
• Describe discrete and continuous action spaces and what clipping does to a command.
• Define an episode and structure a dataset of (observation, action, reward, done) tuples from a rollout.
• List the metadata that must be logged with robot data and say why order must be kept.

The intuition

The state is everything that determines what happens next: where Scout is, where the pillar is, the floor, a person about to step out of a doorway. The observation is what reaches the policy: a few numbers from sensors and a state estimator. Often the observation is a part of the state. If it hides something that matters, the problem is partially observable, and a policy that sees only the current observation may need memory to do well.

The action is the command the policy sends: here a velocity in each of two directions. Actions can be a short list of choices (turn left, go straight, turn right: discrete) or real numbers within limits (continuous). A real actuator saturates, so a command beyond the limit does less than asked, and the log must say whether it stores the command asked for or the one applied.

One attempt at the task, from the start to the end of the run, is an episode. Writing an episode down one step per row gives the table everything else trains on.

The mathematics

The tuple

At each step the logger writes one record. With time steps t = 0, 1, …, T − 1:

o_t

Observation before acting: for Scout [dx, dy, px, py], in metres

a_t

Action applied at step t: [ax, ay], each in [−1, 1], where 1 means 1 m/s

r_t

Reward received after acting (the next lesson defines it)

done_t

True on the last step of the episode, whether by reaching the goal, a collision or the step limit

T

Length of the episode in steps; the time step Δt = 0.25 s fixes the time it covers

A dataset is a collection of episodes. Supervised learning of a policy (module 3) uses the pairs (o_t, a_t); reinforcement learning also uses r_t and the order of the steps; a model-based method uses the successor o_t+1 as well, so many logs store it in a column of its own.

Action spaces

What to log with the numbers

• Frames and units: metres or millimetres, goal minus robot or robot minus goal, map frame or body frame. A sign flip is silent.
• Rates and timestamps: the control period (0.25 s here), and the timestamp of each record, so delays and dropped steps can be found.
• Episode boundaries: an episode id and a done flag, so the end of one run is not read as the start of the next.
• Provenance: who or what chose the action (an expert, a learned policy, a random explorer), the seed and the version of the code.

Order matters

Steps within an episode are not independent: step t + 1 follows from step t. Shuffling single steps across train and test sets lets near-duplicate neighbours leak between them. Split by whole episodes (lesson M2-T6).

A worked example

One episode of Scout, from the start (1, 1). The expert policy drives; the world is deterministic, so the numbers below can be recomputed. The observation is [dx, dy, px, py]; the first three steps and the last are shown.

Reading the first row. The goal (9, 3) minus Scout (1, 1) is (8, 2), so dx = 8 and dy = 2. Scout minus the pillar (5, 3) is (−4, −2), so px = −4, py = −2. The action (0.970, 0.243) is the unit vector towards the goal. Applied for 0.25 s at 1 m/s, it moves Scout by (0.243, 0.061) m to (1.243, 1.061), which is what the next row’s observation shows: 9 − 1.243 = 7.757.

Length and ending. The episode has 34 rows. Step 33 is the last: after it Scout is within 0.5 m of the goal, so the reward is −0.05 + 10 = 9.95 and done = 1. With 33 steps costing −0.05 each the rewards sum to 33 × (−0.05) + 9.95 = 8.30, a number the next lesson uses. The dataset from this one episode has 34 tuples.

Clipping. Suppose a bad controller commands (2, −3) from (1, 1). The world clips it to (1, −1) and Scout moves to (1.25, 0.75), not (1.5, 0.25). If the log kept only the command, a learner would believe that a 2 gave twice the motion of a 1, and would be wrong. Store the applied action, or both.

A dataset of three episodes. Expert runs from (1, 1), (1, 5) and (2, 3.2) have 34, 34 and 32 steps: 100 tuples in all. Tuples per episode vary, so a count of tuples is not a count of demonstrations.

What the simulation simplifies. The observation is the exact relative position, noise-free. It also contains the pillar’s position relative to Scout, so the observation is nearly the full state. Real sensors give a noisy, partial view, and Scout would have to estimate those numbers from lidar or a camera.

Engineering implications and limitations

Write a data card

One page next to every dataset: columns, units, frames, rates, how episodes end, who produced each action, known problems. It takes minutes to write and saves hours when a result looks strange.

Check the log before you train

Plot a few episodes. Check that rewards sum to the expected return, that done appears once per episode and only at the end, that timestamps are evenly spaced and that no row is duplicated. Most dataset bugs show in a plot.

Time limits are not terminations

Ending at the 80-step limit differs from reaching the goal or crashing: the task did not end, the clock did. Many algorithms treat the two differently, so log which one occurred rather than a single done flag when you can.

Mind the sizes

Camera data dominates storage and bandwidth. Decide the rate and resolution the policy needs, store raw frames only if you will reprocess them, and keep a small, versioned sample for tests.

Common misconceptions

Engineering takeaways

1. The observation is what the policy gets; the state is what determines the future. They can differ.
2. Actions are discrete or continuous, and real commands are clipped, so log the action that was applied.
3. A dataset is a set of episodes, each a time-ordered list of (observation, action, reward, done) records.
4. Frames, units, rates, episode ids and who produced the action are part of the data.
5. Split by episode and keep order; a count of tuples is not a count of demonstrations.

References and further reading

The standard textbook and a survey of learning from demonstration for the data view.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018, Chapter 3.
• B. D. Argall, S. Chernova, M. Veloso and B. Browning, “A survey of robot learning from demonstration”, Robotics and Autonomous Systems, 57(5), 2009.
• T. Gebru et al., “Datasheets for Datasets”, Communications of the ACM, 64(12), 2021.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the score attached to each step and the return that adds it up.

Related content

Nothing below is required to finish this lesson.

• Before
  M1-T1: What Is Robot Learning?

  Define a policy and say what it means for a robot to learn, and when learning beats programming and when it does not.

• Lab
  M1-L1: Record a Rollout

  Scout drives a corridor and the log is a mess: turn it into episodes, compute the return, and find the step where the reward is wrong.