Robot Learning · Theory · M1
M1-T1 — What Is Robot Learning?
A policy is a function from what the robot sees to what it does. Robot learning is the set of ways to find that function from data instead of writing it by hand.
Scout, the delivery robot at Meridian General Hospital, must reach a goal at the far end of a hall past a round pillar. You could write the steering rule yourself in ten lines. You could also show Scout a few hundred good runs, or let it try and score each try. All three produce the same kind of object, a policy, and this track is about the second and third routes: how they work, how to tell whether they worked, and why they so often seem to work and do not.
Learning objectives
After this lesson, you should be able to:
- Define a policy as a function from observations to actions and say where the observation and action come from.
- State what is learned, and from what, in imitation learning, reinforcement learning and model-based learning.
- Give two situations where learning is a good choice and two where hand-written control is better.
- Say what this track’s simulated world simplifies and why that is acceptable for learning the ideas.
The intuition
A policy is the robot’s decision rule: given what it currently observes, it says what to do. A thermostat is a policy (temperature in, heater on or off out). So is a PID loop, a behaviour tree with fixed thresholds, and a neural network that maps camera pixels to joint velocities. What differs is how the rule was obtained.
In ordinary robotics an engineer writes the rule, after modelling the robot and the task. In robot learning the rule has free parameters (a table, the weights of a network) and a procedure sets them using data. The data may be examples of good behaviour, a score that says how well each attempt went, or just a record of what happened when the robot did things.
Nothing else about the loop changes. The robot still observes, acts, and the world still responds. Learning is an outer loop that improves the rule between or during runs.
The mathematics
Policy
At step t the robot receives an observation ot and picks an action at. A policy is the rule that does this. It may be deterministic or random:
- ot
- Observation at step t: what the sensors and state estimator give the policy
- at
- Action at step t: the command sent to the robot (a velocity, a torque, a gripper opening)
- π
- The policy; in learning it carries parameters θ, written πθ
- θ
- The parameters that learning sets: a table, weights of a network, gains of a controller
What is learned, and from what
| Approach | Learns | From | Needs |
|---|---|---|---|
| Imitation (behaviour cloning, DAgger) | A policy copying the expert | Demonstrations of (o, a) pairs | An expert, and demonstrations that cover the states the robot will reach |
| Reinforcement learning | A policy, often with a value function | Rewards from the robot’s own trials | A reward, many trials, and a way to reset or simulate |
| Model-based learning | A model of how the world responds, then a plan or policy from it | Logged transitions (o, a, o′) | Enough data for an accurate model; planning at run time |
| Offline learning | A policy from a fixed log | Data collected earlier by someone else | Logged data that covers the actions the policy would choose |
Supervised learning (module 2) is the toolbox underneath: most of these methods fit a function to data. The new difficulty in robotics is that the robot’s own actions decide which data it will see next, so the training data and the situations met after deployment can differ.
When learning helps
- The behaviour is hard to write down. Folding a towel or grasping an object never seen before has no clean equation.
- The input is rich. Mapping raw camera images to actions is a perception problem that hand-tuned rules handle badly.
- Examples are cheaper than models. Showing the task a hundred times can cost less than modelling contact and friction exactly.
When it does not
- A well-understood loop. Holding a motor at a set speed is a job for PID, whose behaviour can be analysed and whose gains can be tuned in an hour.
- Safety must be guaranteed. A learned policy gives no certificate by itself. Where a failure is unacceptable it needs a monitor or a hand-written safe fallback around it.
- Data cannot be had. If you cannot demonstrate, simulate or afford the trials, there is nothing to learn from.
A worked example
Scout’s task, three ways. The world (module simulator) is a 10 m by 6 m hall. Scout starts in the left third, the goal is at (9, 3) and a round pillar of radius 0.7 m stands at (5, 3). Actions are velocity commands in [−1, 1] on each axis, scaled to 1 m/s, applied for 0.25 s per step. The observation is [dx, dy, px, py]: the goal minus the robot, and the robot minus the pillar centre.
| Route | What you provide | What comes out |
|---|---|---|
| Programmed | The rule: head for the goal and swing round the pillar on the side Scout is already on | The expert policy, written by hand. From the start (1, 1) it reaches the goal in 34 steps, 8.5 s |
| Imitation | Expert runs from many starts | A fitted policy that copies the expert on the situations it saw |
| Reinforcement | A reward and many trials | A policy that scores well by its own experience |
What the simulator was used for. For one start, (1, 1), the hand-written expert runs the hall. Its first action is (0.970, 0.243): almost a straight run at the goal, slightly upward, because the vector to the goal is (8, 2), of length 8.25, and 8/8.25 = 0.970, 2/8.25 = 0.243. A policy fitted on raw features [dx, dy] fails to do the same at all: in the labs it reaches the goal from none of the starts, because the expert’s action depends on direction, not distance, and a linear function of dx and dy cannot make that. With unit-direction features a linear policy reaches the goal from nearly every start away from the pillar line, and not from starts near it. The policy class and its features, not the amount of data, decided that.
What the simulation simplifies. Scout is a point-like disc: no heading, no inertia, no wheels slipping, no sensor noise, one obstacle, and the observation contains the exact goal and pillar positions. Real robots have none of these luxuries. That is deliberate: the world is small enough that every result can be recomputed, and the ideas (policy, data, return, failure) are the same.
Engineering implications and limitations
Ask what the baseline is
Before training anything, write the simple controller. If a PID or a state machine already reaches the goal, a learned policy has to beat it on something you care about, such as robustness, speed or effort, before it earns its place.
Learning is a component, not a system
Practical systems keep learned parts inside structure: a learned grasp-scoring network inside a motion planner, or a learned policy under a velocity limiter and an emergency stop. The structure carries the guarantees and the learning carries the flexibility.
Be honest about the state of the field
Learned policies have shown strong results in manipulation with demonstrations, in legged locomotion trained in simulation and moved to hardware, and in some navigation settings. They also need large amounts of data or simulation, break in ways that are hard to predict, and are usually evaluated on few trials. Treat a success rate as an estimate with an error bar (lesson M1-T5), not a fact.
Keep the loop reproducible
Log the seed, the data version and the code version of every run. A result that cannot be rerun cannot be debugged, and robot learning has plenty of results that vanish with the next random seed.
Common misconceptions
“Robot learning means a neural network.”
A policy can be a table, a linear function or a tree. Networks are one choice, useful when the input is rich or the function is complicated.
“A learned policy has learned the task.”
It has fitted the data it saw. On situations unlike that data it may do anything, which is why evaluation on held-out starts matters.
“Learning replaces the engineer.”
It moves the work: choosing observations, actions, rewards, data and evaluation. A badly posed problem gives a confident, wrong policy.
“If it works in simulation it works on the robot.”
Simulators leave out dynamics, delays and noise. Module M10 treats the gap; here the simulator is a classroom, not evidence.
Engineering takeaways
- A policy maps observations to actions, π(o) = a, whether it was written or learned.
- Learning sets the policy’s parameters from data: demonstrations, rewards, or logged transitions.
- The robot’s own actions shape its future data, which is what makes robot learning harder than ordinary supervised learning.
- Learning helps when behaviour is hard to specify or the input is rich; it does not when a simple controller suffices or guarantees are needed.
- The track’s simulated world is simple on purpose: it simplifies sensing, dynamics and obstacles.
References and further reading
Standard surveys and the standard textbook on reinforcement learning.
- B. D. Argall, S. Chernova, M. Veloso and B. Browning, “A survey of robot learning from demonstration”, Robotics and Autonomous Systems, 57(5), 2009.
- J. Kober, J. A. Bagnell and J. Peters, “Reinforcement learning in robotics: A survey”, The International Journal of Robotics Research, 32(11), 2013.
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A thermostat turns the heater on below 20 °C and off above. Is it a policy? What are o and a?
Explanation
Yes. The observation is the measured temperature and the action is heater on or off. It was written by hand, not learned, but it has the same form π(o) = a.
Question 2
Name what is learned, and from what, in imitation learning and in reinforcement learning.
Explanation
Imitation learns a policy from an expert’s demonstrated (observation, action) pairs. Reinforcement learning learns a policy, often with a value function, from rewards earned in the robot’s own trials.
Question 3
A forklift must hold a constant lift speed under varying load. Is learning a good first choice?
Explanation
Usually not. This is a well-understood feedback problem where a PID controller can be tuned, analysed and trusted. A learned policy would need to beat that on something measurable.
What’s next
Next: what the policy sees and does, and how a run becomes a dataset.
Related content
Nothing below is required to finish this lesson.
- LabM1-L2: Learn to Reach the Goal
Fit a simple policy from demonstrations, deploy it, and see it fail from starts it never saw.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.