M9-T5 — Curriculum and Task Design

Learning objectives

After this lesson, you should be able to:

• Describe a start-state curriculum and two other kinds (task difficulty, reward annealing), and state the hypothesis each one makes.
• Compare a curriculum with direct training on the final distribution over many seeds, report the paired difference with an interval, and say when the data cannot tell them apart.
• Compute from the reward the return of the three policies that matter in Scout (reaching, crashing, stalling) and say what a stalling optimum implies for the reward and the task design.
• Explain how a curriculum can hide a failure when scores are reported per stage, and what to evaluate instead.

The intuition

Imagine teaching a delivery robot to cross a hall with a pillar in it. Start it where the pillar is hardly in the way and it learns "head for the goal" quickly, because most of its attempts succeed and the reward is frequent. Then start it directly behind the pillar, where heading straight for the goal means a crash, and it has to learn to curve round. The hope of a curriculum is that the second step is easier than it would have been from scratch, because the first step already taught the robot to move toward the goal. The hope is reasonable, and it has a competitor: the same robot, trained on starts from the whole hall, sees the hard starts from the first iteration and has to learn both things at once from a mixed signal.

Which is better is an empirical question, and the figure is the answer on this task. Twelve seeds, 60 PPO iterations of 12 episodes each, then 100 final-distribution test starts. Direct training (blue) reached 55.0 successes on average, the curriculum (orange) 41.0. Look at the shape and not at the means: in each condition the seeds split into those that learn to reach the goal (84 to 100 successes) and those that score 0. Five seeds (3, 4, 7, 8, 11) are zero in both conditions, two more (6, 9) are zero only with the curriculum, and the successful seeds all score between 84 and 100. The difference between schedules is two seeds. The paired difference is −14.0 successes with a 95% interval from −37.4 to +9.4: no evidence for a benefit, and no evidence for a harm.

The sections below say what is behind that picture, what is honest to conclude from it, and what to do when learning fails the same way on several seeds.

The mathematics

A curriculum is a schedule over distributions

Training samples start states s₀ from a distribution that can change with the iteration t. Direct training uses D_final at every t. A start-state curriculum uses D₁ (easy), then D₂ (near), then D_final:

and what you want to know is a number about the last distribution only:

The regions of the shipped world are: easy, starts with y between 0.5 and 1.3 or between 4.7 and 5.5 (far from the line through the pillar at y = 3); near, y between 2 and 4; all, y between 0.5 and 5.5. x is between 0.5 and 3.5 in all three. All the numbers here need one small change to the trainer: RLCore.trainPolicyGradient takes region as a fixed string for the whole run, and a curriculum changes it during the run. I ran a scratch copy of the trainer in which region may also be a function of the iteration index, which is the only change; everything else (network, PPO settings, seeds, 12 episodes per iteration) is the shipped code. If you run the shipped function, train one stage per call and the network restarts every call, which is not a curriculum.

Other kinds of curriculum

• Task difficulty: fewer obstacles, wider goals, a shorter distance, a lighter payload, then the real task. Same hypothesis as above.
• Reward annealing: start with a dense, generous reward (a progress term, lesson M5-T3) and reduce it toward the real, sparse one, using the potential-based form of lesson M5-T2 so that the final optimum is unchanged.
• Domain-parameter curricula: widen the range of friction, mass or delay as the policy copes (module 10). When the schedule is driven by the learner’s measured success, the curriculum is automatic. Bengio et al. introduced the general idea; Narvekar et al. survey curricula for reinforcement learning.

What a curriculum assumes

• The easy tasks teach something that transfers to the hard ones (a shared skill in the policy), so the learner enters the hard stage in a better place than a random initialisation.
• The learner does not forget or over-specialise on the easy distribution before the next stage arrives.
• The hard task is hard in a way that practice on easier ones can help with, and not because of something else (an unlearnable observation, an unreachable reward).

Each assumption can fail, and the evaluation has to be able to show it.

The reward and the three returns

Scout’s reward is −0.05 per step, +10 for reaching the goal (distance below 0.5 m), −5 for touching the pillar, with an 80-step limit. Three behaviours have a return you can compute without a simulator: reach the goal in n steps, (n − 1)(−0.05) + 9.95 = 10 − 0.05 n; crash after n steps, −0.05 n − 5; stall for all 80 steps, −4.0.

D_final

The start-state distribution you will deploy on (here the region all).

direct

Training from D_final from iteration 0.

paired difference

For seed k, success under the curriculum minus success under direct training with the same seed; the interval is a Student t interval over seeds (lesson M11-T2 has the details).

stall

A behaviour that avoids the pillar by not progressing and runs out the 80-step limit; return −0.05 × 80 = −4.0.

A worked example

All numbers come from Node runs of the scratch trainer described above and of RLCore.evaluate. The evaluation set is 100 starts from RLCore.starts(9001, 100, 'all'), the same for every policy; the policy is the deterministic mean of the actor. Seeds are 1 to 12 (SLAMCore.rng(seed × 77) for the network’s initial weights and exploration noise; the starts of each iteration come from the seed too). Every cell is a mean over 12 seeds unless stated.

By hand: the three returns

The expert reaches the goal from a typical start in 30 steps: 10 − 0.05 × 30 = 8.5, and its mean return over the 100 test starts is 8.51. A policy that drives straight at the goal and crashes after 15 steps gets 15 × (−0.05) − 5 = −5.75. A policy that stands still for 80 steps gets exactly −4.0 (measured: −4.00). The straight-line policy succeeds on 38 of the 100 test starts and crashes on 62; its mean return is −0.13, much better than standing still. Even so, a learner can end at the stall: the stall return −4.0 is better than a crash (−5.75) and worse than any success, so a policy that has learned "the pillar is dangerous" and not yet "how to curve round it" can find that moving less is the best it can do.

The failing seeds are stalls

For direct training, seeds 3, 4, 7 and 8 (60 iterations) gave 0 successes. The measured mean returns of those policies on the test starts were −4.20, −4.09, −4.31 and −4.29, the mean episode lengths 69.9, 74.9, 63.1 and 66.8 steps of a maximum 80, the crashes 14, 7, 23 and 19 of 100, and the mean final distance to the goal 3.7, 4.3, 3.5 and 3.5 m. These are policies sitting in front of the pillar and drifting; their return is the stall return plus a few crashes. The training return over the last five iterations was −4.37, −4.18, −4.25 and −4.39, so the optimisation was not diverging; it had converged to the wrong place. A successful seed (1) scored 100 of 100, mean return 8.37, mean length 32.5 steps.

Curriculum against direct training

Paired differences from direct training (successes out of 100, mean and 95% interval over 12 seeds): curriculum −6.0 (−16.7 to 4.7) at the small budget and −14.0 (−37.4 to 9.4) at 60 iterations; easy then all −1.8 (−11.6 to 8.0) and −7.4 (−26.2 to 11.4); near only +21.8 (−8.4 to 52.0) and +7.0 (−12.1 to 26.1). Every interval includes zero. At 120 iterations with 8 seeds the two schedules scored 48.1 and 48.5 successes (4 of 8 seeds above 80 for both; seeds 3, 4, 7 and 8 at 0 for both), difference +0.4 (−0.9 to 1.6). With these seeds, budgets and this world the data do not show a benefit from the easy-to-hard curriculum, and I should not claim a harm either: the intervals are wide because the outcome is nearly binary per seed.

Two observations that I did not follow up. First, training on near alone was the best schedule in both budgets (60.3 and 62.0), though not significantly; one explanation (unchecked) is that near starts are where the pillar matters, so they carry the most useful gradient, which would be an argument for focusing on hard cases and not for easing in. Second, with the small budget seed 1 failed under direct training and succeeded at 60 iterations, so a failing seed at one budget may simply need more iterations.

How a curriculum can hide a failure

Take the straight-to-goal controller, which cannot avoid the pillar, and report its score the way a stage-wise log would. On 100 easy starts it succeeds 90 times (crashes 10); on 100 near starts 0 times (crashes 100); on the final distribution 38 times (crashes 62). A mixed training log with 90% easy and 10% near starts would show 81 of 100 and read as a good policy. Each number is true and describes a different distribution, and only the third one is the deployment question. The learned policies in my runs did not show this pattern: easy-only training gave 50.6, 51.3 and 51.1 successes on easy, near and all (60 iterations), so the failure was global, not regional. That is a result about this world, in which the failing seeds fail everywhere, and not a reason to stop checking.

Engineering implications and limitations

Checklist for a curriculum

• Evaluate on the final distribution, every time. Keep a fixed test set from D_final (and from each region), never used in training, and report it per stage as well as at the end. A curve that rises at each stage on that stage’s own start states says the learner copes with what it is shown, nothing more (lessons M1-T5 and M11-T2 for the method).
• Report a per-region breakdown (easy, near, all), so that an easy region that improves while the hard one stays at zero is visible. The straight-line example above would be caught immediately.
• Use many seeds and a paired comparison. A mean over 3 seeds of a nearly binary outcome has a spread larger than any plausible curriculum effect; here 12 seeds still had intervals of ±20 successes.
• Compare against the cheap alternatives: direct training with more iterations, a different entropy or step-size setting (lesson M6-T7), a reward with a progress term (lesson M5-T3), or training on the hard region alone. A curriculum has to beat those, not only the weakest baseline.
• Mix, do not switch. Many practical curricula keep a share of the harder or final distribution in every batch, so that the policy cannot forget it; I did not test this.
• Choose the schedule from the learner’s behaviour (advance when the stage success exceeds a threshold, with some hysteresis) and log when each stage ended, because a fixed schedule on a time budget advances stages the learner has not finished.

Task design before curriculum

When several seeds fail the same way, look at the task before the schedule. The stall is a symptom of the reward: it pays −5 for a crash, charges −0.05 per step, and offers +10 only for a success the learner has not yet found, so for a learner that crashes often the safest-looking policy is to hover. Options, none of which I tested on Scout and all of which have trade-offs: a progress term in potential form (lesson M5-T2) that pays for reducing the distance to the goal, which gives a gradient toward the goal even while stalled; a smaller crash penalty relative to the time cost, which makes crashing less frightening and also less costly in the evaluation; ending the episode early on stalling; longer training; or a different start distribution that makes successes common (this is what a curriculum is meant to do). Each changes the learned policy’s behaviour, and the evaluation reward that you report should stay the original task’s.

The same questions apply to any task: what does the policy see as its only route to reward? What is the safest bad policy, and what does it return? What does the time limit do (here: 80 steps, so a stall costs −4.0)? What do the terminations reward (ending the episode by crashing is cheaper than a long stall if the penalty is small)?

Honest limits of this experiment

One world, one network and PPO setting (not tuned), three budgets, a single curriculum split (one third of the iterations per stage), 12 seeds, one fixed set of 100 test starts. The curriculum was a start-state curriculum only. The scratch trainer differs from the shipped one by the single line described in the formulation. A task with real sub-skills (contact, balance, a long horizon) is where curricula are reported to help; this task has a single skill, "curve round a pillar", which the easy region does not contain.

Common misconceptions

Engineering takeaways

1. A curriculum is a schedule over start or task distributions; its hypothesis is that early skills transfer, and it must be tested against direct training on the final distribution, with many seeds and a paired comparison.
2. On Scout (PPO, 12 seeds, 100 test starts), direct training reached 55.0 successes and the easy-near-all curriculum 41.0 at 60 iterations (difference −14.0, interval −37.4 to +9.4); at 120 iterations and 8 seeds they were 48.1 and 48.5. The data show no benefit, and the interval does not show a harm.
3. Most of the variation is between seeds, not schedules: runs end at about 100 or at 0 successes, and the zeros are stalls with returns near the stall value of −4.0, a feature of the reward (crash −5, time −0.05 per step) as much as of the learner.
4. A stage-wise score can hide a failure: the straight-line controller scores 90 on easy starts and 0 on near ones. Always evaluate on the final distribution, and report per region.
5. Look at the reward and the task when several seeds fail the same way: compute the returns of reaching, crashing and stalling, and design the reward and the start distribution so that the best-looking bad policy is not a good place to stop (lessons M5-T2, M5-T3, M9-T6).

References and further reading

Two primary sources for curricula and one for potential-based shaping. The experiments are my own, on a toy world, and are reported with their intervals; they do not establish how well curricula work on real tasks.

• Y. Bengio, J. Louradour, R. Collobert and J. Weston, “Curriculum Learning”, International Conference on Machine Learning (ICML), 2009.
• S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor and P. Stone, “Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey”, Journal of Machine Learning Research 21, 2020.
• A. Y. Ng, D. Harada and S. Russell, “Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping”, International Conference on Machine Learning (ICML), 1999.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The last lesson of the module (M9-T6) takes the reward design question to a legged robot: which observations a gait policy needs, which reward terms it needs, and what each term lets the optimiser get away with. It uses a small gait-parameter model, not a physics simulator, so that the exploits can be counted exactly.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T4: Hybrid Learning and Control Systems

  Place a learned policy inside a classical controller, with a fallback and limits, and say what each part guarantees.

• Lab
  M9-L3: Wrap a Policy in a Safety Controller

  The learned policy is usually good and occasionally terrible: add limits, a monitor and a fallback.