M11-T5 — Designing a Robot Learning Project

Learning objectives

After this lesson, you should be able to:

• Turn a task statement into a learning problem: decision variables, what is learned and what is fixed, data, simulator, metric, hidden test scenarios, budget, and a risk and safety plan.
• Size an evaluation with a zero-failure rule (how many clean episodes demonstrate a success target) and convert episodes into robot time.
• Tie a method choice to the evidence for it in earlier modules (imitation, RL, model-based, offline, transfer), and write the acceptance rule and the go or no-go before training.
• Say when a classical controller already solves the task and learning should not be used.

The intuition

A project that starts with "let us train a policy with PPO" has already made six decisions by accident: the reward, the data, the simulator, the test, the budget and the stopping rule. The accidental ones are the expensive ones to change. A design written first makes each of them an explicit, arguable sentence.

Think of a design as a chain. The task and success criterion say what done means ("a cart of supplies from the pharmacy to Ward 4 in under 6 minutes, with no contact, in at least 95% of runs"). The metric and limits turn that into numbers and hard constraints. The data and simulator are where experience comes from. The method is chosen last, because the first four decide which methods are even possible. Then the right-hand half of the figure, which is where most plans are weakest: an evaluation plan on hidden scenarios, a budget for the robot and people, and a risk and safety plan, ending in a go or no-go written down before any result is seen.

Everything in the chain can be done on the Scout toy of this track, which is why the worked numbers below come from it. The toy is small enough to be exact about each cost, and honest about not being a hospital.

The mathematics

The eight decisions

Sizing the evaluation: the zero-failure rule

If every one of n test episodes succeeds, the 95% Wilson lower bound for the success rate is n/(n + z²) with z = 1.96, so z² = 3.8416. To claim a success rate of at least p with 95% confidence from a clean run, you need

That gives 35 clean episodes for p = 0.90, 73 for p = 0.95 and 381 for p = 0.99 (computed). One failure raises the cost: with one failure, 0.95 needs 110 episodes. Plan the sample size before the evaluation, per scenario family, and note that this assumes the episodes are independent draws from the scenario family, which is a design choice you must defend, not a fact. If an evaluation covers k scenario families and each must meet the bound, the total is k times n.

Budget

For a physical test the cost per episode is the drive time plus the reset time, and the reset usually dominates:

t_drive

Time to complete an episode: the Scout expert takes 30.1 steps on average, 7.5 s at 0.25 s per step.

t_reset

Time to put the robot, cart and environment back for the next episode. I assume 60 s for illustration; it is an assumption, not a measurement.

acceptance rule

The pre-written condition for go: for example, in every scenario family the lower 95% bound of success is at least 0.90 and there are zero hard-limit violations.

pilot test, final test

A pilot test set is for development and may be reused; the final test is run once, after the design is frozen.

Learn or not

Let c₀ be the cost of the classical controller you already have and c_L the cost of building, evaluating and maintaining a learned one, including everything above. Learning is worth considering when the classical system fails a measured requirement, when the learned part addresses a failure the classical one cannot (a perception head, a contact skill, a behaviour that is hard to write down), and when c_L is justified by the benefit. It is not worth it when the classical controller meets the requirement under the same evaluation plan. The comparison has to be on the same hidden scenarios; the baseline that could win (lesson M11-T1) is the classical one.

A worked example

A design: Meridian General's supply robot

The fictional task: a wheeled robot, a bigger Scout, carries a locked supply tote from the pharmacy to the ward stations along corridors with staff, carts and doors. The numbers below are not hospital data; they come from the toy Scout reach task, standing in for "the part we can test cheaply". Every number was computed with RLCore in Node.

Pilot numbers on the Scout toy

The toy answers the cheap part of decision 4: how much demonstration data does the learned part need? Pillar-aware behaviour cloning (featPillar, ridge λ = 0.01) was trained from n expert demonstration starts drawn from the whole start region, for 10 seeds, and scored on a pilot test set of 100 starts per region (starts(901, 100, region)). The acceptance rule used here is "success at least 0.90 in each of the three regions"; it is checked per seed.

Twenty demonstrations were enough in this toy; five were not, and the failures were all in the near region (a result reported in lesson M11-T4). If a real demonstration cost 7.5 s of driving plus 60 s of reset (assumed), 20 demonstrations are 20 · 67.5 s = 22.5 minutes of robot time and 5 are 5.6 minutes: collecting data is cheap compared with the evaluation below. That is a toy statement. A real hospital has people who move and a start distribution I have not modelled.

For RL as the alternative (decision 4), the same toy gives a warning. RLCore.trainPolicyGradient PPO, default settings, 60 iterations of 12 episodes (720 training episodes in simulation), ten seeds, mean policy scored on 100 starts from the whole region: 0.97, 0.82, 0.00, 1.00, 1.00, 0.00, 0.99, 0.00, 1.00, 1.00. The mean is 0.678, the 95% t-interval is 0.34 to 1.015, and an interval above 1 is the symptom of applying a normal-theory method to bimodal data. Three of ten seeds learned nothing in 720 episodes, and no hand-tuned reward would have told me which in advance (lesson M6-T7). The same behaviour cloner needed 20 demonstrations and succeeded in every seed.

Sizing the evaluation and the robot time

The criterion needs a lower bound of 0.95 in each of k = 4 scenario families (empty corridor, one cart, a person crossing, a door opening into the path). With zero failures that is 73 episodes per family, 292 in all; with one failure allowed per family it is 110 per family, 440 in all. At 67.5 s per physical episode (assumed) that is 73 · 67.5 s = 82 minutes per family, or 5.5 hours for 292 episodes, and 8.3 hours for 440. A success target of 0.99 per family would need 381 clean episodes per family, 7.1 hours each and 28.6 hours for four, which is why it is in the budget as an explicit decision and not an afterthought.

In the toy, the classical controller on starts(2001..2010, 100, region) succeeds on 1,000 of 1,000 starts in each of the three regions, a Wilson lower bound of 0.996 in each. It meets this plan by itself. Whether the learned policy is better depends on a requirement the toy does not have; in the toy, the best it does is match the controller it imitates.

Hidden scenarios and the pre-registration checklist

Before any training run, write the following in a file and put it under version control (lesson M11-T3). If you change an item later, record the change and the reason; do not overwrite it.

• Question and criterion: the sentence and the number, with the interval rule (for example the lower 95% bound of success at least 0.95 in every family).
• Hidden scenarios: the families, the generator and seeds for the final test, kept apart from anything used in development. Say what you will never look at before the final run.
• Pilot test: a separate set for development (seed 901 above); the final test uses another seed and is run once.
• Baselines: the classical controller, a random policy, the simplest learner (lesson M11-T1), on the same starts.
• Sample sizes: episodes per family and seeds per method, from the zero-failure rule and the seed variance seen in the pilot (lesson M11-T2).
• Stopping rule: maximum training time or budget, and the rule for abandoning a method.
• Analysis plan: the intervals, the comparisons, the number of tests (to account for multiple comparisons), the weakest-region report (lesson M11-T4).
• Safety limits and rollback triggers: hard limits (no contact, speed near people, forbidden zones), monitors and who may stop the robot.
• Budget and kill criteria: hours of robot and people time, and the point at which the project stops or falls back to the classical controller.
• Go or no-go: the exact outcome that means go, written before the run.

Engineering implications and limitations

When not to use learning

A learned policy is a component with a test burden: data, a simulator, a hidden test, seeds, a monitor, a rollback. If a classical controller meets the requirement under the same evaluation, that burden buys nothing. Reasons to stay classical, with the toy numbers where they exist:

• The controller already works. The hand-written Scout expert reaches the goal on 1,000 of 1,000 starts in each region. A behaviour-cloned copy can at best equal it, and with 5 demonstrations it equalled it in only 4 of 10 seeds; PPO matched it in about 6 of 10.
• The requirement is a guarantee. A learned policy gives a statistical statement from tests (a lower bound after n clean episodes). A controller with a proof or a bounded certificate may give a guarantee for the modelled cases; that is a different kind of claim, and for hard safety limits a learned policy sits behind a classical layer (lesson M9-T4).
• The model is known and the problem is small. Dynamic programming or a planner can solve the Scout grid worlds exactly (lesson M4-T4). Learning approximates what you can compute.
• You cannot test it. If the budget does not allow the 73 clean episodes per family that a 0.95 bound needs, you do not have enough evidence to deploy any policy, learned or not, for that claim.
• You have no data and no simulator. Without demonstrations, logs or a trustworthy simulator, a learning project has nothing to learn from (lessons M8-T2 and M10-T1).
• The maintenance cost is real. Retraining after hardware changes, re-running the test, keeping the monitor calibrated: write them into c_L.

Reasons to learn are equally concrete: a perception problem that no one can write rules for, a contact skill that has no clean model, a classical system that fails a measured requirement in a measured scenario family, or a large amount of cheap data. Even then, learn the smallest part that fixes the measured failure.

Common design mistakes

• The reward is the task. A reward is a proxy (lessons M5-T1 and M5-T4); the success criterion and the safety limits stay in the evaluation, outside the reward.
• One simulator, one test. Randomise for training and keep different hidden worlds for the test (lesson M10-T2, lab M10-L1).
• Tuning on the final test. Every look at the final test turns part of it into training data. Count your looks.
• No fallback. The classical controller is the rollback; keep it running and tested.
• Evaluating the average. The weakest family or region sets the deployment decision, not the mean (lessons M10-T4 and M11-T4).

What these studies do not tell you

The pilot table, the PPO seeds and the budget arithmetic come from a point robot with one pillar, an expert that always succeeds, and no people, no delay and no sensor noise. The timing (60 s reset, 7.5 s drive) is an assumption. The design table is a template with real module citations, not a hospital's validated plan. It shows how the decisions connect, with real numbers where the toy can supply them. It does not show that any of these methods would work on a physical robot in a ward.

Common misconceptions

Engineering takeaways

1. Write the design before training: task and criterion, what is learned, metric and limits, data, simulator, hidden evaluation, budget, and a risk and safety plan, ending in a go or no-go written in advance.
2. Size the evaluation with the zero-failure rule: 35, 73 and 381 clean episodes are needed for 95% lower bounds of 0.90, 0.95 and 0.99; convert them to robot hours, because on hardware the reset time dominates.
3. Tie each choice to evidence from the track: imitation when an expert exists (20 demonstrations were enough in the Scout toy, 5 were not), RL when none does (PPO: 0.678 over ten seeds, three at 0), model-based and offline when real samples are scarce, randomisation and staged rollout for transfer.
4. Use a pre-registration checklist: question and number, hidden scenarios and seeds, pilot against final test, baselines, sample sizes, stopping rule, analysis plan, safety limits, budget, and the go or no-go.
5. Do not use learning when a classical controller already meets the requirement under the same evaluation. These are toy studies on the simulated Scout with no real robot data, so they show how the decisions connect, not what a ward needs.

References and further reading

The decision list and checklist are my own synthesis of the practices in the cited works and the earlier modules, not a standard; the numbers are from scratch runs on RLCore. The hospital is fictional.

• B. A. Nosek, C. R. Ebersole, A. C. DeHaven and D. T. Mellor, “The preregistration revolution”, Proceedings of the National Academy of Sciences 115(11), 2018.
• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mané, “Concrete Problems in AI Safety”, arXiv:1606.06565, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes the methods of the track. Lab M11-L2, the capstone, applies lessons M11-T3 to M11-T5 on Scout: you train and evaluate a policy on hidden scenarios, diagnose where it fails, and write the report.

Related content

Nothing below is required to finish this lesson.

• Before
  M11-T4: Writing an Experiment Report

  Write a report that states the question, the setup, the results, the failures and the limits.

• Lab
  M11-L2: The Learning Capstone

  Train, evaluate and diagnose a policy for Scout on hidden scenarios, then write the experiment report.