When a small language-model agent learns to fix software bugs by imitating a stronger model, it drifts. Each turn it takes slightly off the expert's path moves it into situations the expert never demonstrated, and errors pile up over a long session. The current fix lets the student act for a while, hands control to the expert at a randomly chosen turn, and trains the student on everything the expert does from there to the end. This proposal argues that most of that expert continuation is wasted. The useful signal sits in the few turns right after the student first starts to disagree with the expert, and the rest is either something the student already does well or a long tail the student will rarely reach. We propose to locate that first point of disagreement cheaply, hand control to the expert there, and train only on a short, entropy-filtered window of the expert's correction. The bet is that this matches or beats random-switch correction on SWE-bench Verified while using a fraction of the expert-generated tokens, and that it produces less of the hesitant, looping behavior that unselective recovery data is known to cause.
Two research threads meet here. Interactive imitation learning, in which a learner acts in the environment and an expert labels or corrects the states the learner actually visits, explains why the expert must intervene on the student's own states. Work on redundancy in knowledge distillation, meaning the training of a small "student" model to copy a large "teacher", shows that much of a teacher's dense supervision is redundant once you account for what the student already knows. We use the distillation thread as the selection mechanism and the multi-turn agent setting as the regime where that selection matters most, because each expert token there is expensive to generate and each training token has to compete with the student's limited context.
Core Claim
In multi-turn agent imitation, the value of an expert correction depends on where it starts and on how much of it is kept. The proposal makes three claims, each tested with the expert-token budget held fixed. First, switching control at the student's first high-disagreement turn beats switching at a uniformly random turn. Second, keeping only a short window of the expert's recovery, with tokens where both models are already confident removed, loses little or no downstream accuracy relative to training on the full suffix. Third, re-locating the switch point every round as the student improves beats generating correction data once.
What Prior Work Does and Leaves Open
The method this proposal modifies is On-policy Expert Corrections (OEC). OEC rolls the student out to a switch turn $k$, lets Claude-3.7-Sonnet finish the trajectory, and fine-tunes the student only on the expert suffix. This avoids the per-turn relabeling cost of DAgger (Dataset Aggregation, where an expert labels every visited state), which grows quadratically with trajectory length. OEC chooses $k$ uniformly at random and builds its data in a single offline pass, even though its own analysis shows student drift growing steadily across turns. We keep OEC's roll-in-then-takeover structure and its masking of the student prefix. We replace the random switch with a measured one, cut the suffix down, and iterate.
The distillation thread supplies the selection signal. SEAD runs on-policy distillation on math and sorts tokens by the joint entropy of teacher and student. It skips tokens where both are confident, about half of all tokens, while reportedly retaining most of the training signal, and it uses different divergence objectives for "student unsure" and "both unsure" tokens. DA-KD scores difficulty as the ratio of student loss to teacher loss, selects prompts offline, and finds that training only on extreme-divergence examples destabilizes optimization. What to Distill? (KDAS) makes a related point in vision: samples where teacher and student diverge carry the most knowledge, but only if the teacher is reliable on them. All three work on single-turn outputs or offline data. None uses the disagreement signal to decide when an expert should take over a multi-turn trajectory, which is the step we add. DA-KD's instability result constrains our design: we do not train on the most divergent turns in isolation, but on a window of the expert's own confident continuation that starts at the divergence.
ImitKD first recast sequence distillation as DAgger, with the teacher acting as an oracle on contexts the student generates. It supervises every token, so it motivates our approach without supplying it. On the interactive-imitation side, SafeDAgger trains a classifier to predict when the student's action will deviate from the expert's and queries the expert only then. LazyDAgger adds asymmetric thresholds so that control is not handed back and forth too often. These are the closest conceptual ancestors of a disagreement-triggered switch. Both were built for low-dimensional control, both require training a separate gating model, and both treat everything the expert does during an intervention as equally useful. MTDAgger allocates expert queries across tasks by task need and recent learning progress, but it does not decide anything within a trajectory.
RoTS provides the cautionary result. It synthesizes recovery branches for GUI agents and finds that more than roughly 10% recovery data makes agents over-reflect, second-guessing correct plans. We read this as evidence that unselective recovery data has a real cost. If short, well-placed corrections deliver the benefit of recovery training at lower dose, that cost should shrink. We test this directly.
Research Program
The setting follows OEC. A 7B student is trained on SWE-smith issues and evaluated on SWE-bench Verified. Because our switch signal needs token probabilities from the expert, we use an open-weight 32B coding model as the expert. As a first sanity step, we rerun OEC with this expert so that every comparison shares the same expert.
The switch signal is a per-turn disagreement score. For student turn $t$ with action tokens $a_t$ and context $h_t$, we compute
$$d_t = \frac{1}{|a_t|}\sum_{i} \left[-\log \pi_E(a_{t,i} \mid h_t, a_{t,<i})\right] \;-\; \frac{1}{|a_t|}\sum_{i}\left[-\log \pi_S(a_{t,i} \mid h_t, a_{t,<i})\right],$$
where $\pi_E$ and $\pi_S$ are the expert's and student's next-token distributions. The score is high when the expert finds the student's action surprising and the student does not, which is the "confidently wrong" pattern. It is a log-space analogue of DA-KD's loss ratio, applied per turn. Computing it costs one expert prefill over the student's rollout, which is linear in trajectory length and far cheaper than generating an expert continuation. The switch turn $k^\*$ is the first turn where $d_t$ exceeds a threshold $\tau$. We calibrate $\tau$ once as a quantile of $d_t$ on held-out student rollouts. Rollouts that never cross $\tau$ receive no expert call.
The correction step works as follows. The expert takes over at $k^\*$ and runs to resolution, exactly as in OEC. The student is then trained only on the first $W$ expert turns. Within those turns we drop tokens where both models have low entropy, following SEAD's skip rule. On the remaining tokens we use a bounded objective of the kind DA-KD found necessary for hard examples. The full suffix is still generated, because the expert has to finish the episode for the repetition and resolution filters to apply. What shrinks is the training set: its token count, its dose of recovery content, and its overlap with behavior the student already has.
Training runs in rounds. After each round, new student rollouts are scored and new switch points located. Corrections are therefore always made on states the current student actually reaches, which addresses OEC's single-pass staleness.
The analysis has one central question: is the learning value of an expert correction concentrated near the onset of disagreement? We answer it with a placement sweep that shifts the switch to $k^\*-2$, $k^\*$, $k^\*+2$ and to a random turn, holding the expert-token budget fixed. We also check whether $d_t$ predicts eventual failure better than the student's own entropy does.
Baselines
The naive reference point is behavioral cloning on expert trajectories from SWE-smith, plus the SWE-smith-7B model that OEC reports. It tells us how much any form of on-policy correction buys in our replication.
The strongest existing method is OEC with uniformly random switch turns, training on the full expert suffix, with repetition filtering and prefix masking. OEC reports a 20.8% resolution rate on SWE-bench Verified for OEC-SWE-7B, against 15.2% for SWE-smith-7B (OEC). That number used Claude-3.7-Sonnet as the expert. We therefore treat our own rerun of OEC with the open-weight expert as the operative bar, and report the published figure for context. The comparison is fair only if OEC and our method see the same SWE-smith issues, the same expert, the same number of fine-tuning steps, and the same hyperparameter search budget. We run it at two budgets: one equal number of expert-generated tokens for both methods, and OEC's natural budget. Beating it means a higher resolution rate at equal expert tokens, or the same resolution rate at substantially fewer expert and training tokens. In both cases we report across several training seeds, because single-run differences of a point or two on Verified are within noise.
A learned-gate baseline adapts SafeDAgger: we train a small classifier on student hidden states to predict whether the expert would disagree, and switch when it fires. This tests whether our training-free score is merely a cheap stand-in for something a trained classifier would do better.
Three ablations isolate the ingredients. The first keeps our switch point but trains on the full suffix, isolating the windowing. The second keeps random switching but applies our window and token filter, isolating the placement. The third replaces $d_t$ with the student's entropy alone, removing the expert prefill cost and testing whether the expert's view matters. A matched-size random token drop on OEC data checks whether entropy filtering beats plain subsampling.
Why not just…?
Why not just run OEC with more random switches, since expert calls are cheap compared with human labels? This is the real threat. If placement does not matter, more random corrections at equal cost could match us. The budget-matched comparison and the placement sweep settle it directly. If resolution rates are flat across $k^\*\pm2$ and random placement at equal expert tokens, the placement claim is false. We would then report the windowing and filtering result on its own, as an efficiency finding.
Why not just truncate the expert suffix to a fixed length or drop tokens at random? SEAD reports that its entropy zones could not be replaced by uniform reduction and that vanilla on-policy distillation suffered entropy collapse (SEAD). That evidence comes from single-turn math, however, so it may not transfer. The random-drop ablation at matched token count tests whether it does in multi-turn coding.
Why not just keep the student's successful rollouts (rejection sampling)? OEC reports that this underperforms, because successful but loopy long rollouts teach bad habits (OEC). We include it as a cheap reference, but we do not expect it to compete.
Why not just train a SafeDAgger-style classifier to pick the switch? It might work. The learned-gate baseline is in the study for that reason. Our score needs no labeled training data and adapts automatically as the student changes between rounds, whereas a classifier must be retrained each round. If the classifier wins, the contribution narrows to the windowing and iteration.
Proposed Experiments
The primary evaluation reports SWE-bench Verified resolution rate for the 7B student, plotted against expert-generated tokens and training tokens, across multiple seeds. It also reports two behavioral measures: mean turns per resolved issue, and the fraction of rollouts caught by repetition filtering. These are our proxy for over-reflection in the sense of RoTS.
Four checks are decisive. The placement sweep must show resolution peaking near $k^\*$ at fixed budget. The full-suffix-versus-window ablation must show that windowing costs at most a small amount of accuracy while cutting training tokens substantially. The iterated-versus-single-pass comparison must favor iteration. And $d_t$ must predict eventual task failure better than student entropy alone; if it does not, the cheaper entropy variant should replace it.
For a secondary check of generality, we apply the same switch-and-window recipe to GUI agents. We use GUI-RobustEval, where RoTS reports 26–33% success at error depth 5 (RoTS), and test whether disagreement-located corrections hold up as error depth grows. A positive result in both domains would support a general rule for agent distillation: pay the expert to correct the student where the two first disagree, and train only on that part of the correction.
References
- Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
- What to Distill? Fast Knowledge Distillation with Adaptive Sampling
- Autoregressive Knowledge Distillation through Imitation Learning
- Query-Efficient Imitation Learning for End-to-End Simulated Driving
- LazyDAgger: Reducing Context Switching in Interactive Imitation Learning
- Data-Efficient Multitask DAgger
- Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents