When a small language model learns from a large one, the student writes its own answers and the teacher corrects them. Researchers call this on-policy distillation. It trains the student on the kinds of mistakes it will actually make at test time, but early student drafts are often poor. The best current fix, AdaSwitch, lets the student write until its predictions drift too far from the teacher's, then hands the rest of the sequence to the teacher for good. We think this one-way handoff has a hidden cost. On long reasoning chains, the handoff tends to happen early, so the late part of most training sequences is written by the teacher. The student then almost never practices continuing on its own from a corrected state, which is exactly where errors compound at test time. The late tokens become off-policy (written by a model other than the student) again, and so they reintroduce the mismatch between training and test conditions that on-policy training was meant to remove.
We propose a two-way switch. The teacher takes over when the student's divergence spikes, and it hands control back once the student, reading the teacher's recent tokens, would again predict what the teacher predicts. The controller borrows a pattern from adaptive search in swarm optimization, where good methods move repeatedly between wide search and local refinement based on feedback rather than a clock. We add hysteresis, meaning two separate thresholds for switching in each direction, like a thermostat, so control does not flicker from token to token. The claim is concrete and testable. At equal compute, two-way switching should beat one-way switching, and the gain should be concentrated on long sequences and late positions.
Core Claim and Mechanism
At decoding position $t$ we compute $d_t = \mathrm{KL}(p_T(\cdot \mid x_{<t}) \,\|\, p_S(\cdot \mid x_{<t}))$, the divergence between the teacher's next-token distribution $p_T$ and the student's $p_S$. The distillation loss already needs both distributions at every position, so this costs no extra forward passes. The controller is in one of two states, student-writing or teacher-writing. It moves from student to teacher when $d_t > \theta_{\text{up}}$. As in AdaSwitch, $\theta_{\text{up}}$ is a sliding-window mean of recent divergences plus $k$ standard deviations. It moves back to the student when the moving average of $d$ over the last $m$ teacher-written tokens drops below a lower threshold $\theta_{\text{down}} < \theta_{\text{up}}$, and only after a minimum dwell of $L$ tokens or at a natural boundary such as the end of a reasoning step. A cap on switches per sequence bounds fragmentation. The loss is unchanged from AdaSwitch: a generalized KL on every token, whoever wrote it. Only the choice of who writes which token changes.
The ideas come from different places. The one-way threshold controller and its evaluation setup come from AdaSwitch, which motivates the gap by stopping at a single switch. The two-way, feedback-driven design is borrowed from metaheuristic search. There, cyclical cosine control factors such as the one in Li et al.'s improved barnacle optimizer and in Cao and Lin let a search re-enter exploration after it has settled. Distance-based inertia rules like Jiang's (high momentum far from the goal, low momentum near it) tie that choice to measured error rather than elapsed time. We use divergence as the "distance to goal" and drop the clock entirely. We also include a clock-driven cosine variant as an ablation, to test whether feedback matters.
Several related works set limits on this design. AdaSwitch's own argument that fragmented "chimera" sequences hurt training is a real risk, and it is why we add hysteresis and a dwell time rather than switching at every token. SPOT and RLAD balance exploration against teacher guidance through verifier rollouts or reward-shaped trust regions. Both need an outcome reward. Our controller needs none, so it applies to summarization as well as math. AdaKD adapts which tokens receive loss and at what temperature, whereas we adapt which model generates each token. The two approaches are complementary and do not compete. Like all of these methods, ours requires teacher logits, so it does not address black-box teachers.
Research Program
We will build the controller on top of a reimplementation of AdaSwitch, sharing its code path. With both switching thresholds set equal and the switch cap set to one, the controller reduces exactly to AdaSwitch. Before any tuning, we will run a diagnostic on AdaSwitch alone. For each training sequence, we will record where the handoff occurs and plot the fraction of student-written tokens against relative position. If that fraction is not substantially lower in the last third of long GSM8K solutions than in the first third, the motivating premise fails and the project should stop there. If the premise holds, we will train two-way variants and analyze three things: accuracy, where in the sequence the accuracy gains come from, and whether the switched training sequences are coherent enough to learn from.
Baselines
The naive reference points are two. The first is supervised fine-tuning on teacher-generated outputs, which is fully off-policy. The second is pure on-policy distillation, where the student writes everything and receives teacher logits as targets. Together they bracket the problem AdaSwitch was designed to solve.
The strongest existing method is AdaSwitch itself. It is the published method closest to our question and reports the best results on our target benchmarks. According to its paper, it beats the second-best method on GSM by up to 11.8% for the Llama model and is best overall across summarization, GSM, and GSM-Plus. The file gives only relative gains, so we found no reported absolute AdaSwitch accuracy or Rouge-L on Lacuna and will reproduce these numbers ourselves. All methods will use the same student–teacher pairs from the Qwen 2.5 and Llama 3.1 families used by AdaSwitch, the same training prompts, the same budget of generated tokens and teacher forward passes, and the same number of hyperparameter configurations searched on a held-out validation split. We count beating AdaSwitch as a gain over three seeds that exceeds seed variance on GSM8K and GSM-Plus accuracy, together with no loss on DialogSum Rouge-L.
The most dangerous simple competitor is random-prefix handoff. The teacher writes a prefix of random length, the student continues, and the process can alternate at random points. This also puts student tokens late in sequences, but without any divergence signal. We also include a clock-driven cosine oscillation between writers, which is the direct import from metaheuristics. The ablations that isolate our key ingredient are two-way switching without hysteresis (both thresholds equal, no dwell time) and two-way switching with the switch cap set to one, which is AdaSwitch.
Why not just…?
Why not just let the teacher correct each divergent token and immediately return control? This is the limiting case of our method with zero dwell time and no hysteresis. AdaSwitch argues that it produces chimera sequences that train poorly. We include it as an ablation. If it matches the full method, hysteresis is unnecessary. If it underperforms, that supports AdaSwitch's argument and our design choice.
Why not just mix random teacher prefixes with student continuations? This is a genuine threat. It would recover late-position on-policy training at almost no complexity cost. The decisive comparison is made at a matched on-policy token fraction by position. If random handoff matches our controller once that fraction is equalized, the benefit comes from the positions where the student writes, not from adaptive control, and we will report the simpler method as the result.
Why not just use a fixed cosine schedule, as the swarm-optimization literature does? Clock-based oscillation ignores whether this particular student, on this particular prompt, has recovered. Jiang's distance-based rule suggests feedback should do better, but the evidence in the file comes from path planning, not language models. The cosine ablation settles the question directly.
Why not just skip the teacher and run reinforcement learning with a verifier? On math, RLAD reports that plain GRPO reaches 48.5% Pass@32 on AIME25, while teacher-anchored updates reach 66.4%. That result suggests teacher guidance still helps. More fundamentally, DialogSum has no verifier, and we want a method that works without one.
Proposed Experiments
The main benchmarks are GSM8K, GSM-Plus, and DialogSum, following AdaSwitch. We add MATH-500 as a longer-sequence stress test, since longer chains should amplify any late-position effect. The metrics are accuracy and Rouge-L. We will also report accuracy bucketed by the number of steps in the reference solution, and the student-written token fraction by relative position.
Three decisive checks follow. First, the gain over AdaSwitch should grow with solution length. It should be near zero on the shortest bucket and largest on the longest. A flat gain across buckets would falsify the late-exposure explanation even if the average improves. Second, the method must beat random-prefix handoff at matched positional on-policy fraction; otherwise adaptivity adds nothing. Third, as a coherence check, we will train on switched sequences and verify that the rate of malformed student outputs and the scores on DialogSum do not degrade relative to AdaSwitch, which would indicate that chimera fragments are being learned. All results will be reported over three seeds with identical compute accounting, including wall-clock overhead from the extra writer switches.
References
- AdaSwitch: Adaptive Switching Generation for Knowledge Distillation
- Equivalent Circuit Modeling and Parameter Identification for Lithium-ion Batteries Based on Improved Barnacle Mating Optimizer
- Fault Diagnosis Based on Optimized Wavelet Packet Transform and Time Domain Convolution Network
- Path planning with obstacle avoidance for soft robots based on improved particle swarm optimization algorithm
- SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
- Reinforcement-aware Knowledge Distillation for LLM Reasoning
- LLM-Oriented Token-Adaptive Knowledge Distillation