When a robot or a language-model agent learns by imitating an expert, it drifts into situations the expert never demonstrated. Small mistakes then pile up. Interactive imitation learning handles this by letting the learner act, then having a teacher take over and show what to do from the states the learner actually reaches. A recent cost-saving trick is to let the teacher take over for only a few steps, $K$, rather than finishing the whole task. Every such method has to choose $K$, and the obvious way to choose it is to try a few values and keep the one that most improves the learner after the next retraining round. This note argues that this procedure is myopic in the same way that tuning a learning rate on a 100-step window is myopic. It favors corrections that repair frequent, shallow mistakes quickly. It starves rare, deep mistakes, where the real cause lies many steps before the visible failure and only long corrections help.
The proposal has two parts. The first is a diagnosis: over several rounds of data collection, the value of $K$ that looks best after one round differs systematically from the value that gives the best final learner. This gap should appear only under conditions that mirror those known from meta-optimization, namely noisy rollouts combined with a mix of shallow and deep error types. The second part is a remedy. We build a planner that allocates the teacher's budget across correction lengths over all remaining rounds, using a cheap model of how each kind of correction reduces each kind of error. The key test is simple. The same planner with a one-round lookahead should reproduce the bias, and the multi-round version should remove it.
Where this sits
The phenomenon comes from meta-optimization. Understanding Short-Horizon Bias in Stochastic Meta-Optimization shows that tuning a learning rate by its effect over a short window drives it far too low. On their 1,000-dimensional noisy quadratic, the greedy schedule stalls at a final loss of 63.86, while the long-horizon schedule reaches 4.25. The cause is that small steps calm the steep, noisy directions of the loss surface but leave the flat directions untouched. They also show that the bias vanishes without both noise and ill-conditioning. That condition is the one we transfer. Related failures appear in learned optimizers (Understanding and correcting pathologies in the training of learned optimizers) and in greedy population-based training (To Be Greedy, or Not to Be – That Is the Question for Population Based Training Variants). MetaOptimize fixes the bias by optimizing a discounted sum of future losses. It works only for continuous, differentiable hyperparameters, and it has never been applied to imitation. We borrow its objective but not its gradient machinery, because $K$ is discrete and its effect passes through data collection.
The imitation work supplies both the problem and the competitors. A Few Teacher Steps Go a Long Way introduced OP-SHORT, a two-phase scheme for language-model agents. The student first runs on its own for a while (the "roll-in"). The teacher then takes over for only $K$ steps. All of these continuations are kept, whether or not the task succeeds. The paper reports that $K=3$ beats $K=5$ on HotpotQA. It also reports that with 10% of the demonstration data, OP-SHORT matches full demonstrations plus reinforcement learning on Terminal-Bench-Dev. The authors state that $K$ is a hand-tuned constant, that they ran only a single round, and that short continuations fail when the cause of an error lies about 20 steps upstream.
That paper motivates the proposal, and it is not contradicted by it. We predict that short $K$ is correct for one round and early rounds, and wrong as a fixed policy across many rounds. SCoRe attacks deep errors differently: the teacher locates and fixes only the first wrong step. This requires a teacher able to localize errors. ThriftyDAgger decides when a human steps in and hands back control using two signals: disagreement among an ensemble of policies, and a learned estimate of the chance of reaching the goal. It uses fixed thresholds, with a stricter threshold for handing back control than for taking it. It optimizes safety within the current episode, not the learning value of a correction. DART avoids on-policy rollouts by injecting noise into expert demonstrations. It constrains our design, because its evidence that full DAgger rollouts are costly and unsafe is the reason full-length corrections are not a free default.
Core Claim
Every learner failure can be labeled by its root-cause depth $d$: the number of steps between the first wrong action and the point where recovery becomes impossible. Such labels can be computed by replaying teacher continuations from earlier prefixes, which is the same operation OP-SHORT already performs.
A correction of length $K$ mainly teaches the learner to fix failures with $d \lesssim K$. Short corrections are cheap, so many can be collected, and they quickly reduce frequent shallow failures. This makes them look best after one round. Deep failures are rare and noisy to observe, so a one-round evaluation gives them almost no weight. Once shallow errors are exhausted, deep errors dominate the remaining failure rate.
This is the structure of the noisy quadratic. Shallow errors play the role of steep, high-variance directions, and deep errors play the role of flat ones. The prediction is therefore specific. Greedy selection of $K$ is biased toward short horizons. The bias grows with the spread of root-cause depths and with rollout stochasticity, and it disappears when all errors are shallow or rollouts are deterministic.
Research Program
The method is a budget planner over rounds $r=1,\dots,R$. At each round, it tracks the learner's closed-loop failure rate $f_d$ separately for each depth bucket $d$. It also maintains a small learning-curve model $g_d(n_{d,K})$, which predicts how $f_d$ falls as the number $n_{d,K}$ of length-$K$ corrections covering depth $d$ grows. This model is refit online from observed round-to-round changes. The planner chooses how to split the teacher's per-round budget $B$ across correction lengths so as to minimize the discounted future failure rate:
$$\min_{\{n_{r,K}\}} \sum_{r'=r}^{R} \gamma^{r'-r} \sum_d p_d\, f_d^{(r')} \quad \text{s.t.} \quad \sum_K c_K\, n_{r',K} \le B,$$
Here $p_d$ is the share of episodes that reach a depth-$d$ decision, $c_K$ is the teacher cost of a length-$K$ correction, and $\gamma$ is a discount factor. We solve this by receding-horizon search over a small discrete set of $K$ values. After each round, we execute only the first round's allocation and then replan.
Setting the planning horizon to one round yields the greedy twin. The greedy twin uses the same model, the same data, and the same budget, and differs only in foresight. This mirrors the greedy-versus-long-horizon comparison in Ren et al.
Around this core we build three artifacts. The first is a synthetic chain environment in which we directly set the distribution of root-cause depths and the transition noise, serving as the analogue of the noisy quadratic. The second is a root-cause labeling tool that reuses prefix replays. The third is a multi-round OP-SHORT harness, which fills the single-round gap that the OP-SHORT authors leave open.
Baselines
The naive reference is behavior cloning on the initial demonstrations, trained with the same total gradient steps as all other methods.
The strongest existing method is OP-SHORT from A Few Teacher Steps Go a Long Way, run for $R$ rounds with its reported best fixed $K$ and its 50–90% on-policy mixing ratio. The file contains no absolute accuracy for OP-SHORT on HotpotQA or ALFWorld (no reported result found on Lacuna for those numbers). The only quantitative anchor is that it matches 100% demonstrations plus reinforcement learning on Terminal-Bench-Dev using 10% of the demonstrations. We therefore rerun it ourselves.
The honest upper reference is the best fixed $K$ chosen by complete $R$-round runs for every candidate $K$. This costs several times our compute, and we report that cost.
We also include three further baselines. The first is a hand-designed schedule in which $K$ grows linearly across rounds. The second is a per-episode handoff rule in the style of ThriftyDAgger, which ends the correction when a learned goal-probability estimate crosses a threshold. The third, on reasoning tasks, is SCoRe-style first-error correction.
The ablation that isolates our contribution is the greedy twin planner. For the continuous-control task, the external anchor is ThriftyDAgger's reported 73% autonomous success on simulated Robosuite peg insertion. This compares against 48% for LazyDAgger and 24% for SafeDAgger. Our comparison there uses an algorithmic expert and the same intervention budget.
Fairness rules apply to every method. All methods share the same student initialization, number of rounds, teacher-token or intervention budget, and gradient steps per round. Each gets the same tuning budget for its own hyperparameters, and we run at least five seeds.
We count as beating the field three results together. First, the planner must exceed fixed-$K$ OP-SHORT and the greedy twin in final success, with confidence intervals excluding zero. Second, it must come within noise of the exhaustive best-fixed-$K$ oracle at a small fraction of its cost. Third, it must beat that oracle on tasks where the depth mix shifts across rounds.
Why not just…?
Why not just grid-search $K$ with full multi-round runs? This is the real threat, because it is unbiased by construction. Our answer is cost, plus one structural limit. A single fixed $K$ cannot follow a mix of error depths that shifts across rounds, while a schedule can. If the exhaustive oracle matches the planner on every task, the practical contribution shrinks to a cheaper tuning method, and we will say so.
Why not just hand the learner back control when a risk critic says it is safe? A ThriftyDAgger-style critic measures whether the learner can continue the current episode. It does not measure what the correction will teach over later rounds. Its thresholds are also fixed quantiles, which the ThriftyDAgger paper itself leaves open. We run it as a baseline. If it matches the planner, then episode-level safety is a sufficient proxy and our long-horizon argument fails.
Why not just use full corrections, as in standard DAgger? DART reports that DAgger was about 280% slower in computation on Humanoid. It also reports that DAgger's training policies earned 80% less reward than the supervisor. OP-SHORT reports that longer continuations can hurt. Full corrections therefore waste budget on shallow errors, and they appear as the $K=\infty$ point in our sweep.
Why not just let the teacher find the first error, as SCoRe does? Localizing the first error addresses deep errors directly, and on reasoning tasks it may make long corrections unnecessary. SCoRe reports a 52.3% reasoning average, compared with 43.0% for behavior cloning. It depends on a teacher that can localize errors, which is unavailable in continuous control. We compare against it on the reasoning benchmarks. If it wins there, we will restrict our claim to domains where error localization is unavailable.
Why not just increase $K$ on a fixed schedule? If a linear ramp does as well, then the diagnosis is the finding and the planner adds little. We treat that outcome as a legitimate result.
Proposed Experiments
The first experiment tests the diagnosis in the synthetic chain. We sweep the spread of root-cause depths and the transition noise. For each setting we measure two quantities: the gap between the greedily chosen $K$ and the long-horizon-optimal $K$, and the resulting gap in final failure rate. The decisive check is the one Ren et al. found for the noisy quadratic. Both gaps should be large only when depth spread and noise are both present, and near zero when either is removed. If the gap is uniform across conditions, the proposed mechanism is wrong.
The second experiment runs agentic benchmarks for multiple rounds: HotpotQA, ALFWorld, and Terminal-Bench-Dev, following OP-SHORT, with $R=5$ rounds and a fixed teacher-token budget. The metrics are final task success, success broken down by root-cause depth, and teacher tokens spent. The prediction is that the greedy twin reproduces OP-SHORT's preference for short $K$ early on. The multi-round planner should then shift budget toward longer corrections in later rounds and improve success specifically on deep-error episodes.
The third experiment runs continuous control. We use Robosuite peg insertion and the RoboMimic Square task with scripted experts, under intervention budgets matched to ThriftyDAgger. The metrics are autonomous success and intervention steps.
Across all experiments, we also check whether the depth-bucket learning-curve model predicts next-round failure rates to within their seed variance. If it cannot, the planner's gains could come from extra exploration rather than foresight. We would test that by comparing against a planner that uses randomized allocations of equal entropy.
References
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Understanding and correcting pathologies in the training of learned optimizers
- To Be Greedy, or Not to Be – That Is the Question for Population Based Training Variants
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
- A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training
- Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- DART: Noise Injection for Robust Imitation Learning