Asking for Help Only When the Robot Is Ignorant, Not When It Has a Choice

Robots that learn from human demonstrations are short on data, and every new demonstration costs a person's time at a teleoperation rig. One appealing fix is to let the robot run on its own and hand control to a human only at moments when it is unsure. Modern imitation policies, however, are built to be unsure in a harmless way. Diffusion and flow policies are generators that turn random noise into action sequences. They deliberately spread probability over several valid behaviors, such as going around an obstacle on the left or on the right. Any trigger based on the spread of their sampled actions will keep calling the human to choice points the robot has already mastered, which wastes the scarce budget the method was meant to save.

This note proposes a different trigger. We train a small ensemble of cheap action generators and ask whether the ensemble members agree on the distribution of actions, not whether the sampled actions agree with each other. Two members that both put half their mass on "left" and half on "right" agree perfectly, even though their samples differ wildly. We measure agreement with an unbiased two-sample distance between members' action samples. We calibrate the handover threshold on held-out successful rollouts, and we feed the human corrections back into the policy online. The claim is concrete: for a fixed amount of human teleoperation time, this trigger yields higher task success than state-novelty triggers and naive variance triggers on tasks where demonstrations are genuinely multimodal. It should not lose anything on tasks where they are not.

Core Claim

Classical active learning treats predictive variance as a sign of missing data. Two results support this view. Dropout-based Active Learning for Regression uses the spread across 25 dropout forward passes as an estimate of what the model does not know. Active learning for logistic regression: an evaluation shows that variance reduction gives a reliable floor where error-seeking heuristics can fall below random selection.

That reading breaks for generative robot policies. AdaFlow trains a network to predict the conditional variance of a flow policy's actions and finds that this variance peaks at multimodal decision states. AdaFlow uses the signal only to decide how many integration steps to take at test time. Its variance therefore describes spread in the demonstrations, the irreducible kind, and not model ignorance. In the terms of active learning, it is aleatoric rather than epistemic.

The closest system that triggers human help from uncertainty is PATO (also Assisted Teleoperation for Scalable Robot Data Collection). It handles the conflation by hand. A latent-variable subgoal model's sample variance flags "which task next" choice points, and disagreement among an ensemble of recurrent low-level policies flags unfamiliar states. Fixed thresholds then decide when to call the human. PATO thus treats task multimodality as a reason to ask for help, and it measures low-level disagreement as spread of point predictions.

Our proposal differs in three ways. It uses a single score that is near zero exactly at well-learned multimodal states. It applies to diffusion or flow action chunks rather than recurrent point predictors. And it closes the loop by retraining on interventions rather than only harvesting a static dataset.

The score is defined as follows. Let $\pi_1,\dots,\pi_M$ be the ensemble members, each a conditional action generator. Draw $K$ action chunks $a^{(i)}_1,\dots,a^{(i)}_K$ from member $i$ at observation $s$. The epistemic score is the mean pairwise energy distance between members:

$$ E(s) = \frac{2}{M(M-1)} \sum_{i<j} \Big[\, 2\,\overline{\|a^{(i)}-a^{(j)}\|} \;-\; \overline{\|a^{(i)}-a^{(i)\prime}\|} \;-\; \overline{\|a^{(j)}-a^{(j)\prime}\|} \,\Big], $$

Each bar is an average over sample pairs. Within-member terms are averaged over distinct pairs only, which makes them U-statistics and the estimate unbiased.

The expected value of $E(s)$ is zero when all members represent the same distribution, however many modes that distribution has. It grows when members disagree about which modes exist or how much weight each mode gets. This is the property naive approaches lack. The total sample variance is dominated by within-member spread. Disagreement among member means is also inflated: with $K$ samples, a mean's sampling noise scales with the within-member variance divided by $K$, so it is largest exactly at multimodal states.

Research Program

The first component is an affordable ensemble. Full ensembles of diffusion transformers are too slow for a 10–20 Hz control loop. We therefore attach $M=5$ small flow-matching action heads to one shared visual encoder. Each head gets a different initialization and a bootstrap resample of the demonstration data. Following AdaFlow, the heads are trained so that one to a few integration steps suffice, which makes drawing $K=8$ samples per head cheap. The score is computed once per predicted action chunk rather than per control tick, which cuts the rate by the chunk length.

A shared encoder can make members agree for the wrong reason. To check this, an ablation compares the shared-trunk ensemble against five fully independent policies on a subset of tasks, and reports both latency and trigger quality.

The second component is the handover rule. We borrow the calibration idea from Uncertainty-Aware Failure Detection for Imitation Learning Robot Policies, which fits a time-varying conformal band to scalar uncertainty signals on successful in-distribution rollouts. Conformal calibration sets the threshold from held-out data so that false alarms on successful runs occur at a chosen rate $\alpha$. That work only detects failures; it does not ask a human for help or learn from the result. We use the band as the trigger. Once triggered, we add the minimal-hold mechanism of RND-DAgger: the human keeps control for at least a fixed number of steps.

The band must be recalibrated after each retraining round, because the policy changes. Without recalibration the guarantee no longer holds. The same failure-detection paper reports that fixed calibration degrades under environment shift.

The third component is the loop. Human segments are appended to the dataset. Heads are refit from the shared encoder every $N$ interventions, and the encoder is refit less often. The question the program answers is how quickly success rises per minute of human time.

Baselines

The naive reference point is passive collection at the same human budget. A person teleoperates full demonstrations for the same total number of expert-controlled steps, and a Diffusion Policy (Diffusion Policy) is trained on them. Beating it means a higher success rate at equal expert steps. A second reference point is random handover at the same intervention rate as our trigger. It separates the value of the score from the value of interleaving autonomy with intervention.

The strongest state-based competitor is RND-DAgger. It hands control to the expert when a predictor network fails to match a fixed random network on the current observation, which is a measure of state novelty. RND-DAgger reports reaching peak performance three times faster in expert time than Ensemble-DAgger on 3D maze navigation. That result was obtained on low-dimensional and raycast tasks, and we know of no result on visuomotor manipulation. The score is a single forward pass, and Uncertainty-Aware Failure Detection reports that RND scores run 36–44x faster than STAC, so it is the speed-matched threat.

The strongest action-based competitor reimplements PATO's two-signal logic on our policy class. It uses sample variance for task ambiguity and spread of member means for execution uncertainty. PATO reports 82% intervention precision at decision points versus 48% for ThriftyDAgger, and a throughput of 15.88 demonstrations versus 13.06 for ThriftyDAgger, in its real kitchen study.

Two ablations isolate our key ingredient. Both use the identical ensemble, loop, and conformal calibration, and change only the score. One uses total sample variance. The other uses disagreement among member means. Any difference from our method is then attributable to the energy-distance score alone.

All triggers get the same encoder, the same training compute per round, and the same hyperparameter search budget. All are compared at matched intervention rates, set by choosing each method's $\alpha$. To beat a baseline, our method must achieve higher final success at the same expert-step budget, averaged over at least three seeds, with non-overlapping confidence intervals on the multimodal tasks.

Why not just…?

Why not just use state novelty, as RND-DAgger does? State novelty is cheap and cannot be fooled by multimodal actions, because it never looks at actions. Its weakness runs the other way. It fires on visual change, such as distractors or lighting, when the motor skill is unaffected. The failure-detection study above finds that RND-style scores degrade under environment shifts like crumpled towels and distractors. It also stays silent in familiar states where the policy has learned only one of two required modes. We test both failure directions with distractor-perturbed evaluation scenes and with tasks whose second mode is under-represented. We also run an "either signal" combination, because the two scores may be complementary rather than competing.

Why not just collect more demonstrations or synthesize them in simulation? Synthetic scaling helps but stalls. RoboCasa lifts atomic-task success from 28.8% to 47.6% with 3,000 MimicGen demonstrations, yet reports composite tasks below 12% and real-world success of 24.4% after co-training. Passive data also over-represents nominal trajectories and under-represents the recovery states a deployed policy drifts into. Our passive baseline at matched budget tests this directly. MimicGen-style synthesis is orthogonal and could seed the initial dataset for every method.

Why not just use the sample variance of the diffusion policy, or a single network's predicted variance as in AdaFlow? This is the real threat. If benchmark tasks are nearly unimodal, all variance signals coincide and our extra machinery buys nothing. We therefore evaluate on standard tasks and on deliberately multimodal variants, and we predict a tie on the former and a gap on the latter. A single-model variance head has a further problem. Its outputs in unseen states are as unconstrained as the policy itself, so it cannot signal ignorance. Our variance-score ablation covers this case.

Proposed Experiments

The main study runs in simulation with scripted experts, so that we can run many seeds and counterfactual rollouts.

The first task family is Push-T, a planar pushing task that is multimodal and where Diffusion Policy reports 95% real-world success. The second is RoboMimic Square and Transport. The third is LIBERO-10 from LIBERO. For context on LIBERO, AdaFlow reports 91% average success across the suite against 88% for Diffusion Policy at 20 steps. No reported result found on Lacuna for intervention-based active collection on any of these benchmarks.

For each family we also build a multimodal variant in robosuite. The scripted expert randomly picks one of two valid routes or placement targets with fixed probabilities, so the true action distribution has known modes.

Each run starts from 10 demonstrations and allows a fixed budget of expert-controlled steps. We report four measurements. The first is success rate against expert steps. The second is intervention precision: the fraction of handovers after which a counterfactual autonomous continuation from the same simulator state would have failed. The third is the false-trigger rate at known choice points. The fourth is per-chunk latency.

The decisive checks follow from the claim. On the multimodal variants, our score should trigger rarely at choice points once both modes are learned, while the variance ablations keep firing. Our success curve should dominate both ablations and RND-DAgger at equal budget. On unimodal tasks the gap should vanish. If the gap appears there too, the explanation is something other than multimodality and must be understood.

Ensemble collapse is the main way the score could be confidently wrong. We test for it by withholding one mode entirely and checking whether the members still disagree. Finally, a small real-arm study with two operators on a two-placement-target task repeats the matched-budget comparison against RND-DAgger. It also measures how much operator time each trigger saves per successful demonstration.