Teaching Robot Policies From Their Rollouts: Budgeted Demonstration Generation Guided by a Learned Competence Map

Robot learning is short on data, and the main ways around the shortage are open-loop. Simulation pipelines take a few dozen human demonstrations and generate thousands of variations without asking whether the policy being trained already handles those variations or cannot yet learn from them. Machine teaching studies how a teacher should choose training examples to move a learner toward a goal. Most of its theory assumes an all-seeing teacher that can read the learner's parameters and update rule. Modern robot policies break that assumption: they are diffusion models or billion-parameter vision-language-action models whose weights reveal nothing useful about which skills they have. What a teacher can do cheaply is watch the policy try a task and record whether it succeeds.

We propose a teacher that observes only rollout outcomes. From sparse trial runs it maintains a small predictive model of where the policy succeeds across task configurations, which we call a competence map. It spends a fixed simulation budget on new demonstrations wherever that map predicts the largest gain per unit of cost. Unlike existing interactive robot learning, the teacher decides what to teach, not the robot. Unlike existing black-box teaching methods, it does not need policy likelihoods and does not need to probe the whole input space. The concrete claim is that on RoboCasa, this teacher matches the success rate of uniform 3,000-demonstration generation with roughly a third of the simulator budget, with the cost of the trial runs counted against it.

Core Claim

Three bodies of work each stop short of this design. Black-box machine teaching shows that interaction is necessary. Teaching a black-box learner proves that a teacher who never queries the learner may have to label the entire domain, while an interactive teacher comes within a logarithmic factor of the optimal teaching set. Towards Black-box Iterative Machine Teaching shows that a teacher can rebuild a "virtual learner" from exam responses and nearly match an all-seeing teacher. These papers motivate our work, but they also constrain it. Both assume either learners that fit every training example perfectly or linear learners, both use low-dimensional static classification, and Dasgupta et al. require probing the whole instance space. None of these conditions holds for a visuomotor policy trained by stochastic gradient descent.

The closest robotics-facing work is Curriculum Design for Teaching via Demonstrations. It ranks expert demonstrations by how unlikely they are under the learner's current policy and reports self-correction when earlier skills are forgotten. It needs the learner's action likelihood over whole trajectories, and it was tested in grid worlds and a synthetic driving simulator. For a diffusion policy, that likelihood is expensive or ill-defined, so we keep its teacher-driven structure and replace the likelihood with rollout outcomes.

Interactive imitation methods such as ThriftyDAgger and Fleet-DAgger do allocate scarce expert effort. In both, however, the request comes from the robot, based on ensemble disagreement or learned failure risk in the state it currently occupies, and nobody holds a model of the policy's competence across tasks. We treat these as the main competing family, not as prior art we extend.

What would be new is a teacher that estimates competence from success and failure outcomes alone, shares that estimate across related configurations so that probing stays sparse, and selects demonstrations by predicted learning gain rather than by failure rate. All of this happens under a single budget that charges for both probing and generation.

Research Program

The setting is RoboCasa (RoboCasa), which pairs 50 human demonstrations per atomic task with a MimicGen-style generator inside MuJoCo. Its 120 kitchen scenes and 153 object categories let us partition each task into configuration cells. A cell is a task crossed with a group of object categories and a group of scene layouts. We alter the generator only so that it samples new demonstrations from a cell the teacher chooses. The learner is Diffusion Policy (Diffusion Policy), which generates actions by iteratively denoising random noise. As a second learner, we use LoRA fine-tuning of OpenVLA, a 7-billion-parameter model that maps images and language to actions. The teacher never sees the weights, gradients, or action likelihoods of either learner.

The teacher has three parts.

The first part is the competence map. It is a logistic model over cell features with a Gaussian-process correction, fit to a running record of trial rollouts. Sharing across cells lets a failure on one mug-in-cabinet configuration lower the estimated success on similar configurations without testing them. This is our substitute for exhaustive probing, and the map handles the policy's noisy, inconsistent behavior through a posterior over success rather than hard elimination of hypotheses.

The second part is a gain model. It is fit online to the history of how much a cell's success, and the success of its neighbors, moved after each batch of demonstrations was added there. The teacher picks the next batch by

$$c^* = \arg\max_c \frac{\mathbb{E}[\Delta S \mid k \text{ demos in } c] + \beta\,\sigma_c}{\text{cost}_c},$$

where $\Delta S$ is the predicted change in success averaged over the held-out exam distribution, $k$ is the batch size, $\sigma_c$ is the competence map's posterior uncertainty for cell $c$ (a bonus that pays for probing poorly understood cells), $\beta$ weights that bonus, and $\text{cost}_c$ is the simulator steps needed to generate and filter $k$ successful demonstrations in $c$. Expensive cells, where generation often fails, are penalized naturally.

The third part is scheduled re-testing of cells the map marks as mastered. This catches forgetting, the behavior Yengera et al. observed, without depending on likelihoods.

The denominator of the selection rule is the honest part of the design. The budget is total simulator environment steps, and probe rollouts and generation attempts both draw from it. A teacher that wins only by running many trial episodes for free has not won anything.

We will also measure, as an analysis result in its own right, how many rollouts per cell the competence map needs to rank cells by learning gain reliably. This is the robotics counterpart of the open question in black-box teaching of how much probing interaction must cost.

Baselines

The naive reference point is the 50 human demonstrations per task with no generation. The standard reference is RoboCasa's own pipeline, which generates demonstrations uniformly across configurations. Nasiriany et al. report that adding 3,000 generated demonstrations to 50 human ones raises atomic-task success from 28.8% to 47.6% (RoboCasa). We rerun this pipeline with our learner, seeds, and evaluation configurations, and use it at matched budgets of 500, 1,000, and 3,000 generated-demonstration equivalents. Beating it means reaching at least 47.6% (or our reproduced uniform-3,000 number, whichever is higher) at no more than 1,000-equivalent total budget including probes, or exceeding it significantly at equal budget.

The strongest active competitor is a disagreement-gated selector in the style of ThriftyDAgger. It trains an ensemble of five policies and generates demonstrations in the cells with highest ensemble action variance on sampled initial states. Optionally it adds a learned risk estimate, as in the original. ThriftyDAgger reports 73% autonomous success on robosuite peg insertion against 48% for LazyDAgger, so this is the method a reviewer would expect to be hard to beat. It pays no rollout cost but pays five times the training compute. We therefore report both a step-matched comparison and a total-compute-matched comparison, and we give it the same hyperparameter search budget of 20 trials as our teacher. Beating it means higher success at equal simulator steps and no worse at equal wall-clock compute.

Three ablations isolate the key ingredient. A failure-weighted teacher allocates demonstrations in proportion to estimated failure rate, which removes the gain model. An independent-cells teacher uses no sharing across cells, which removes the sparse-probing mechanism. A teacher without re-testing removes the forgetting check. As an upper bound standing in for an all-seeing teacher, a dense-probe oracle evaluates every cell with 50 free rollouts before each allocation. The gap between our teacher and this oracle measures what opacity plus sparse probing costs.

Why not just…?

Why not just generate more data uniformly, as RoboCasa does? Uniform generation is the baseline we must beat, and it improves atomic tasks substantially. But it leaves composite tasks with success often below 12% (RoboCasa), and NLP evidence suggests targeting the learner's frontier can beat full-data training with 35% of the labels (Feeding What You Need by Understanding What You Learned). That evidence comes from another domain, so it is suggestive only. The step-matched learning curves will settle the question. If uniform generation at 1,000 demonstrations is within noise of our teacher, the proposal's main claim fails.

Why not just put more demonstrations wherever the policy fails? This is the most serious threat, because it captures most of the intuition in a single line. Our bet is that the lowest-success cells are often ones the policy cannot yet learn from, where generation is also expensive, and that the gains come from cells of intermediate success and their neighbors. The failure-weighted ablation tests exactly this. We will also report the rank correlation between predicted and realized gain for both rules. If failure weighting matches the full teacher, we will report that the competence map's value lies only in sparse estimation, and we will drop the gain model.

Why not just use ensemble disagreement, which needs no rollouts? Disagreement is cheap to compute, but it measures uncertainty in actions, not task success. It reacts only to the states sampled, and it cannot detect a skill that the policy has confidently forgotten. The ThriftyDAgger-style baseline, run under both step-matched and compute-matched accounting, decides whether rollout-based competence estimates are worth their cost.

Why not just fine-tune a pretrained VLA and skip teaching? Pretraining helps but leaves reliability incomplete: OpenVLA reaches 70.6% on BridgeData V2 (OpenVLA), and the pretrained model still needs data for adaptation. We therefore treat the backbone as a separate factor and run every allocation rule on top of OpenVLA fine-tuning as well. If the teacher's advantage disappears with a pretrained backbone, that is a meaningful negative result about when teaching matters.

Proposed Experiments

The primary experiment covers all 25 RoboCasa atomic tasks. It plots success against total simulator steps for each allocation rule and each backbone, with three seeds and 50 evaluation episodes per task on held-out scenes. As a secondary test, we run the five composite tasks with the highest reported success. There, cells are defined per subtask, and the teacher may allocate across subtasks of the composite.

The decisive checks are as follows. The first is whether the teacher reaches uniform-3,000 performance at no more than 1,000-equivalent budget with probes included. The second is whether it beats the failure-weighted ablation, which tells us whether learning gain rather than failure carries the signal. The third is the size of the gap to the dense-probe oracle, which tells us how much opacity costs once sharing across cells is used. The fourth is whether re-testing measurably reduces drops in success on previously mastered cells.

For transfer beyond RoboCasa, we run the same teacher on robosuite peg insertion with a scripted expert in the role of the human. There we compare against ThriftyDAgger's reported 73% autonomous success, counting expert-generated steps as the budget. No reported result was found on Lacuna for any teacher-driven demonstration selection method on RoboCasa or with diffusion or vision-language-action learners, so those comparisons will be established by our own reruns.