M8-T2 — Dataset Coverage and Quality

Learning objectives

After this lesson, you should be able to:

• Define state-action support and coverage, and describe concentrability in words.
• Measure coverage with a count table and with the fraction of an optimal (or candidate) policy’s pairs that appear in the data.
• Separate the quality of a dataset (return of the behaviour) from its coverage, and show with measured numbers that more data and better behaviour do not guarantee more coverage.
• Recognise how narrow logs, mixed logs and logging bias hide failures, and write the data-card items that expose them.

The intuition

Imagine two logs of the same hospital hall. The first was written by a very good driver who took the same corridor 60 times, with a few slips. The second was written by a clumsy driver who wandered everywhere, mostly failing. Which log do you want to learn from? The driver of the first log gets a high score. Yet nearly every turn the first driver did not take is missing from the first log, and the missing turns are exactly where a learner, trying to improve on the driver, will look.

The figure shows this in numbers for Scout’s ward. The groups are ordered from the most expert behaviour on the left (almost always the optimal action, ε = 0.1) to the most random on the right (ε = 0.9). The second line under each group is what share of the optimal policy’s own state-action pairs the log contains. For the expert logs it is 49%: the driver’s corridor, and nothing beside it. For the random logs it is 98%. The naive learner, which trusts unseen actions, does worst exactly where the behaviour is best and coverage is poorest. The conservative learners are much less sensitive but do not escape, and every learner is better on the broad log.

Neither "good behaviour" nor "much data" is the property the learner needs. It needs coverage of the decisions that matter.

The mathematics

Support, coverage and concentrability

Write μ(s, a) for how often the dataset contains the pair (s, a), as a fraction of all transitions, and d_π(s, a) for how often the policy π would visit that pair in the real world (its discounted visitation).

support

The set of pairs with μ(s, a) > 0: the choices the data show at least once. In a continuous space "at least once" becomes "near enough to a logged pair", and where near enough ends is a modelling decision.

coverage of a policy π

The share of π’s pairs that lie in the support. The coverage of the optimal policy is what the table below measures; coverage of a candidate policy is what you can measure on a real robot.

concentrability

Roughly, how much more often π visits a pair than the data do: the largest ratio d_π(s, a) / μ(s, a). If π visits something the data never contain, the ratio is infinite. Analyses of offline learning typically need it to be bounded; I use it only as a description of the idea and do not use a bound in this lesson.

behaviour quality

The expected return of the policy that wrote the log. It says how good the logged episodes are. It says nothing about the pairs the behaviour did not choose.

Measuring coverage

1. Count table: for every (s, a) count the transitions. Rows of zeros are states the log never entered; a row with one non-zero entry is a state where the log shows a single action.
2. Coverage of a policy: take the policy you care about (optimal in a simulation, the candidate in a real study), look up each of its pairs in the count table, report the fraction with count ≥ 1 and the list that fail.
3. Low-count states: report states with fewer than two distinct actions, where an offline learner has no comparison to learn from (lab M8-L2 builds exactly these three functions).
4. Continuous states: replace the table by distances to the nearest logged pair, a density model, or the disagreement of an ensemble of value functions. These are indicators, not guarantees, and the threshold that separates "covered" from "not covered" is a choice you must document.

Quality: the kinds of behaviour policy

Benchmark suites name datasets by the behaviour that produced them. The D4RL benchmark of Fu et al. (2020) uses the names random (a uniformly random policy), medium (a partly trained policy), expert, medium-expert (a mixture of the last two) and medium-replay (the replay buffer of an agent while it was learning to medium level). The names describe quality. As this lesson shows, they describe coverage only indirectly, and two datasets with the same name can differ in coverage because of how they were collected.

A worked example

Every number comes from running Node against RLCore: the ward (slip 0.2, step reward −0.1, goal +10, stairwell −10, discount 0.95; optimal value at the start 3.31), collectDataset with the optimal policy plus ε-random actions, offlineQ with 80 sweeps and step size 0.3. Learners: naive with q0 = 10, behaviour-constrained ("seen") and penalised (target − 1/√count); both of the latter with q0 = 0. Scores are exact values of the greedy policy at the start (evaluatePolicyDP), mean of 20 seeds. The ward has 23 non-terminal states and 92 state-action pairs.

A count table (seed 1, ε = 0.2, 60 episodes, 952 transitions) for the states on the optimal route, with the counts for up / right / down / left:

Reading it: on the route the optimal action is covered about 80 times. Beside it, the other actions are covered a handful of times each (the epsilon-random moves and the slips). In total, 11 of the 23 non-terminal states appear in the log at all, so 11 of the 23 optimal pairs are covered (48%), and 42 of the 92 pairs (46%). The other 12 states, including the middle and south-west corridors, have rows of zeros. In state 19 the log shows up and left but neither right nor down.

The size by randomness table. For each of 12 combinations: the number of logged episodes, the behaviour’s randomness ε, the mean number of transitions, the share of the optimal pairs in the log, the behaviour’s own mean discounted return (the log’s quality), and the three offline values. The exact expected return of the behaviour policy itself is 2.70 at ε = 0.1, 2.04 at ε = 0.2, −0.09 at ε = 0.5, −2.03 at ε = 0.9 and −2.07 for a fully random policy (exact values; the sampled means in the table differ slightly).

Five readings. (1) Quality and performance run the opposite way: the ε = 0.9 logs have the worst behaviour (−2.03) and give the best naive policy (2.43 at 120 episodes). (2) Size without coverage does little: at ε = 0.1 the number of transitions quadruples from 417 to 1,664 and the coverage of optimal pairs stays at 48% to 50%; the naive value stays at −2.00; the constrained learner rises from 0.27 to 1.48, which is the benefit of better estimates on the pairs the log does contain. (3) Covering the optimal pairs is not enough for the naive learner: at ε = 0.5 with 60 episodes the log contains 86% of them, and the naive value is still −1.93, because the naive max also reads the pairs around the route, and only 69% of all pairs appear. (4) Conservatism buys robustness: the constrained learner stays between 0.27 and 2.46 and the penalised learner between 1.33 and 2.72 across the 12 rows, while the naive learner swings from −3 to 2.4. (5) Almost nothing reaches the optimum here: the best cell is 2.72 of 3.31 and the next best 2.46, and 80% of the optimum is 2.65. For the constrained and penalised learners the standard deviation across seeds of a cell is between 0.8 and 3.0, so differences of a few tenths between neighbouring cells are within noise.

"But I can collect more." At ε = 0.2 with 500 episodes (about 7,800 transitions) the optimal pairs in the log rise to 73% (mean of 20 seeds), and with 2,000 episodes (about 31,000 transitions) to 95%. At ε = 0.1 they only reach 55% at 500 episodes (6,926 transitions) and 64% at 2,000 episodes (27,672 transitions), with the least covered seed at 52% in both. The naive value at ε = 0.1 stays −2.00 at both sizes. So a log of nearly 28,000 transitions can still miss a third of the optimal policy. The least covered seed at ε = 0.2 is also 52% at 2,000 episodes, while the best is 100%, which shows that coverage depends on the chance of one slip into a side branch. I did not run a log of 100,000 transitions.

Mixed and replay-style datasets (60 episodes in total each; "constrained" and "penalised" as above; the behaviour return is the sample mean over all logged episodes, including time-outs):

A little random data is not a cure for the naive learner: with 6 random episodes among 54 good ones it was worse (−3.08) than with none, because it adds some unseen actions but not enough to make the max honest. Half and half is adequate for the conservative learners (2.31 and 2.28), which are close to their best here. "Medium-replay" is my stand-in for the D4RL dataset of that name: five batches whose randomness falls from 0.9 to 0.1, as if from a learner improving; it is not the benchmark itself. The constrained and penalised learners did well on all the mixtures that contained a broad part.

What a narrow log hides. In the ε = 0.2 logs, 85% of the logged episodes reach the goal and the mean return is 2.22, a log that looks successful. The policy that the naive learner extracts from it reaches the goal in 0% of simulated episodes. Only about 9 transitions per log (95% interval 7.3 to 10.4) end in the stairwell, and the log shows 51% of the states not at all. The log’s success rate describes the behaviour policy, not any policy learned from it. I also tried the survivorship-bias story: dropping all failed episodes from an ε = 0.5 log (71% of the episodes are kept). I expected worse policies. The exact values did not get worse (constrained 1.85 to 2.34, penalised 2.34 to 3.31, the optimum), but the simulated stairwell rate of the deployed policies rose from 4% to 18% and from 5% to 11% (the optimal policy itself falls in the stairwell in about 10% of episodes). I did not trace why, so I do not claim the mechanism; the lesson is that a value metric alone did not show a change in risk, which is the point of lesson M5-T5.

What this simulation simplifies. A tabular, 23-state world in which coverage is exactly countable and the optimal policy is known; on a robot you only have candidates, and coverage is approximate. One behaviour family (epsilon-random around the optimum), one learner family, 20 seeds.

Engineering implications and limitations

The data card for an offline dataset

Lesson M2-T6 asked for a data card for every dataset. For a log to be used offline, add these items:

Practical advice

• Add deliberate variety when you can. A small share of randomised or noisy actions in safe regions (the ε in the example, or injected noise) widens coverage at small cost, if the robot can tolerate it. In the table, ε = 0.5 gave 86% coverage of the optimal pairs at 60 episodes against 49% for ε = 0.2.
• Log the failures. Keep near-misses and interrupted runs and mark them. A policy that never sees a stairwell edge cannot be told to avoid it.
• Report coverage for the candidate policy, not only for the dataset. "The dataset covers 90% of states" is vacuous if the policy’s route is in the missing 10%.
• Do not equate the benchmark names with coverage. A dataset called "expert" may be exactly as narrow as the ε = 0.1 logs above.
• Re-measure after you change anything in the observation, the action clipping or the reward.

Common misconceptions

Engineering takeaways

1. Quality is the return of the behaviour; coverage is which state-action pairs the log contains. They are different, and coverage is what the learner needs.
2. Measure coverage with a count table, the fraction of the candidate policy’s pairs that appear, and a list of the states with fewer than two logged actions.
3. On the ward, the ε = 0.1 logs covered 48% to 50% of the optimal pairs at 30, 60 and 120 episodes, and the naive value stayed −2.00; the ε = 0.9 logs covered 95% to 100%, and gave 0.81, 1.71 and 2.43.
4. More data helps only if it adds coverage: 27,672 transitions at ε = 0.1 still covered 64% of the optimal pairs.
5. Mixed datasets help the conservative learners more than the naive one, narrow logs hide failures, and the data card should record the behaviour, the selection, the action logged and the coverage report.

References and further reading

Fu et al. introduced the D4RL benchmark and its dataset names; Levine et al. discuss coverage and the distributional-shift problem of offline RL; Gebru et al. is the origin of the data-sheet idea behind the data card.

• J. Fu, A. Kumar, O. Nachum, G. Tucker and S. Levine, “D4RL: Datasets for Deep Data-Driven Reinforcement Learning”, arXiv:2004.07219, 2020.
• S. Levine, A. Kumar, G. Tucker and J. Fu, “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, arXiv:2005.01643, 2020.
• T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III and K. Crawford, “Datasheets for Datasets”, Communications of the ACM, 64(12), 2021 (arXiv:1803.09010).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

We have seen that the naive learner fails where coverage is poor, and that the conservative learners do better. The next lesson, M8-T3, shows why: a max over actions in the Bellman target trusts the value of actions the data never contain, and the error compounds through bootstrapping.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T1: Learning from Fixed Datasets

  Define offline learning and the conditions in which robot logs can be used to train a policy.

• Lab
  M8-L2: Find the Coverage Gap

  Which states and actions does the log never show? Measure the coverage and predict where the policy will fail.