M1-T4 — Training Versus Inference

Learning objectives

After this lesson, you should be able to:

• List what is learned (parameters) and what is chosen by people (architecture, features, reward, simulator, hyperparameters).
• Contrast training and inference by where they run, what they may use and what limits them.
• Explain train, validation and test sets and why the test set is used once.
• Compute a latency and memory budget for a policy that must run at 20 Hz.

The intuition

Think of a driving instructor and a driver. During lessons the instructor corrects, repeats, stops the car and tries again. On the test the driver is alone, the road is real, and what was learned is all there is. Training is the lessons. Inference is the test drive.

In a learned policy, almost everything is fixed before training starts: the shape of the model, the features it sees, the reward, the simulator, the step size. Training changes only one thing, a vector of numbers θ called the parameters. When training ends, θ is written to a file. That file, plus the code that evaluates it, is the policy that goes on the robot.

One thing makes this harder than it sounds. In ordinary supervised learning the data are fixed. In robotics the policy chooses the actions, the actions decide which states the robot visits, and the visited states are the data. A different policy sees different data. That loop is why a policy that looks fine on the workstation can fail on the floor, and it is taken up in lessons M1-T6 and M3-T3.

The mathematics

A policy as a function with parameters

o, a

observation and action; for Scout o = [dx, dy, px, py] and a = [ax, ay] in [−1, 1].

θ

the learned parameters: weights of a linear map, or of a neural network.

L

a training objective, for example the mean squared error between the policy’s action and the expert’s.

θ*

the parameters when training stops; the only thing deployed.

What is learned and what is fixed

Training and inference compared

Train, validation, test

Data are split into three parts. The training set is used to fit θ. The validation set is used to choose everything else: features, model size, step size, when to stop. The test set is used once, at the end, to report a number. Each time a choice is made by looking at a score, a little information leaks from that set into the choice, so a set that has guided choices is no longer an honest test. For robots the split is by episode or scenario, never by single time step: neighbouring steps of one run are nearly identical, and splitting them would test on near-copies of the training data (lesson M2-T6).

Frozen or continual

Most deployed policies are frozen: they change only when a person ships a new version that has passed evaluation. A policy that keeps learning on the robot can adapt, but it can also forget, drift, or learn from its own mistakes with nobody to check. If it learns online, the safety filter and a fixed fallback behaviour still have to work without it.

A worked example

A 20 Hz budget. At 20 Hz the control period is 1 / 20 = 0.050 s = 50 ms. Everything from reading the sensors to the motors getting the command must fit, and the time that matters is the slowest step, not the average, because one late command is one step of driving on old information.

The model. A small network 4 → 64 → 64 → 2 has weights and biases 4·64 + 64 + 64·64 + 64 + 64·2 + 2 = 320 + 4160 + 130 = 4610 numbers. In 32-bit floats that is 4610 × 4 = 18 440 bytes, about 18 kB. A forward pass needs 4·64 + 64·64 + 64·2 = 4480 multiply-adds, about 9000 operations, a few microseconds on any CPU. In Scout’s world the expert rule itself takes about 0.3 microseconds per call when timed in Node. Compute is not the problem for this task.

A camera policy. Suppose a vision policy has 25 million parameters and needs 5 GFLOP per step. In 32-bit floats it is 25 × 10⁶ × 4 = 100 MB. On an on-board accelerator that delivers 200 GFLOP/s in practice, one step takes 5 × 10⁹ / 2 × 10¹¹ = 25 ms. Added to the other stages it would give 6 + 25 + 2 + 8 = 41 ms typical and 10 + 25 + 3 + 12 = 50 ms worst case: no margin, so any hiccup misses a deadline. The options are to run at 10 Hz (100 ms), shrink the model, reduce the numerical precision, or process the image at lower resolution. Which one is acceptable depends on how fast the hall changes: at Scout’s 1 m/s top speed a 50 ms delay is 5 cm of travel, and a 100 ms period is 10 cm.

What the simulation simplifies. RLCore has no latency at all: the action chosen for step t acts on the state it was computed from. On the real robot the observation is already old by the time the action is applied. Lab M1-L2 only trains and runs a linear map; the budget above is arithmetic about a deployment the track does not perform.

Engineering implications and limitations

Freeze, version and test the artifact

Deploy a file, not a notebook: parameters, the exact feature code and the normalisation constants, with a version number. A common silent bug is training with one scaling of the inputs and running with another, which gives a policy that is “broken” by a factor of ten.

Check training and deployment agree

Feed a few logged observations to the deployed code and compare the actions with those from the training code. They must match to rounding error before anything else is tested.

Keep a safety layer that does not learn

A speed limit, a clearance check and an emergency stop should not depend on θ. They still work when the learned part misbehaves, and they are the part regulators and colleagues can read.

Measure latency where it counts

Time the whole loop on the robot, with the real sensors and a loaded computer, and report a high percentile such as the 99th, not the mean. Keep the policy’s input and output timestamps in the log.

Use the test set once

Write down what “the test” is before training, keep it out of tuning, and run it at the end. If it must be reused, say so and treat the number as optimistic.

Common misconceptions

Engineering takeaways

1. Training chooses the parameters θ; everything else is fixed by people beforehand.
2. Inference runs the frozen θ on the robot against a clock, a memory limit and a safety filter.
3. Validation is for choosing, the test set is for one final report, and robot data are split by episode.
4. Latency is a budget with a worst case: 20 Hz leaves 50 ms for sensing, policy, safety and actuation.
5. The policy changes the data it sees, so training results do not carry over automatically to deployment.

References and further reading

Standard textbook treatments of learning from data and of the agent and environment loop.

• R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction”, 2nd ed., MIT Press, 2018 (Chapter 1 and the agent-environment interface).
• T. Hastie, R. Tibshirani and J. Friedman, “The Elements of Statistical Learning”, 2nd ed., Springer, 2009 (Chapter 7, model assessment and selection).
• S. Ross, G. Gordon and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, AISTATS, 2011.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: how to measure whether a policy is good, with intervals that say how sure the number is.

Related content

Nothing below is required to finish this lesson.

• Before
  M1-T3: Rewards, Returns and Objectives

  Write a reward for a task, compute the discounted return of an episode, and say what the reward does not say.

• Lab
  M1-L2: Learn to Reach the Goal

  Fit a simple policy from demonstrations, deploy it, and see it fail from starts it never saw.