TL;DR
Selective Residual Adaptation (SRA) is a plug-and-play framework that boosts the reasoning performance of small, distilled Large Language Models (LLMs) without requiring a teacher model at inference time. By identifying sparse "divergence points" and re-ranking the student's top candidate tokens using a lightweight adapter, SRA achieves significant gains in mathematical reasoning (up to 4.37% Pass@16) while keeping the base model frozen.
What Problem Does This Paper Address?
Knowledge distillation is the standard way to shrink massive LLMs into "student" models that are cheap to run. However, these smaller models often hit a performance ceiling, especially in complex, multi-step reasoning tasks like math or coding.
While "collaborative inference"—where a small student generates most tokens but asks a large teacher for help at difficult steps—can close this gap, it’s a deployment nightmare. It requires keeping a massive teacher model in memory and introduces significant latency. The authors ask a compelling question: Can we "internalize" these teacher corrections into a lightweight mechanism that operates independently?
Methods
The authors started with two key empirical observations:
- Sparsity: The student only diverges from the teacher at a small fraction of positions (roughly 14%).
- Candidate Coverage: Even when the student picks the wrong token, the teacher's preferred token is almost always sitting in the student's "top-K" list.
Based on this, they developed SRA, which consists of two modules added on top of a frozen student model:
- Discrepancy Gate (DG): A tiny MLP that monitors the student's internal hidden states. It predicts whether the current token is a "divergence point" where the student is likely to stray from the teacher's logic.
- Candidate Residual Adapter (CRA): If the gate triggers, this module (a small bidirectional Transformer) looks at the student's hidden states and the embeddings of the top-K candidate tokens. It predicts "logit residuals"—small adjustments to the scores—to re-rank those candidates and steer the student back on track.
Training happens in two stages: a "Cold Start" phase using offline student trajectories, followed by an "On-Policy" phase where the adapter is refined on trajectories generated by the newly enhanced policy to handle distribution shift.
Datasets
- Training: The models were trained on DAPO-Math-17k, a dataset focused on mathematical reasoning.
- Evaluation: The primary benchmarks were math-heavy: AIME24/25, AMC23, and MATH-500.
- Generalization: To ensure the model didn't lose its general abilities, the authors also tested on GPQA-Diamond (science), HumanEval (coding), and MMLU-Pro (multidisciplinary).
Key Insights and Ideas
- TV Distance > KL Divergence: The authors found that Total Variation (TV) distance is a much more sensitive indicator for training the Discrepancy Gate than standard KL divergence. TV distance grows linearly with small probability shifts, making it easier for a small MLP to detect when a top-1 token is about to flip.
- Internalizing Collaboration: SRA essentially acts as a "cached" version of a teacher's wisdom. Instead of calling the teacher, the student uses the CRA to simulate what the teacher would have done at that specific difficult step.
- Preserving Diversity: Because SRA only intervenes at selected positions, it avoids the "mode collapse" often seen in standard fine-tuning, preserving the model's ability to explore different solution paths (crucial for metrics like Pass@K).
Key Results
The researchers tested SRA on the Qwen3 family (0.6B to 4B parameters).
- Consistent Gains: SRA improved the average Pass@16 across nearly all settings. For the Qwen3-4B model, it raised the average Pass@16 by 4.37 percentage points over the distilled baseline.
- Beating the Teacher: In some cases, like AIME25, the SRA-enhanced 4B model actually surpassed the 8B teacher model it was learning from.
- Overcoming Plateaus: The authors showed that SRA provides better gains than simply continuing standard distillation for more epochs, suggesting it captures nuances that global loss functions miss.
Limitations and Future Work
- Top-K Assumption: SRA relies on the teacher's preferred token being in the student's top-K. While this holds true for structured reasoning, it might be less reliable in open-ended, creative generation where the vocabulary space is less constrained.
- Training Overhead: While inference is very efficient, training requires on-policy rollouts and teacher forward passes, which is computationally expensive.
- Scope: The current work focuses heavily on structured reasoning (math/code). The authors note that exploring how this applies to free-form dialogue is a key next step.
Summary updated