SRA: Selective Residual Adaptation for LLM Reasoning Enhancement

Zhenxing Dou, Kun He, Mingyang Yi, Yong LiuarXiv 2026 · arXiv

TL;DR

Selective Residual Adaptation (SRA) is a plug-and-play framework that boosts the reasoning performance of small, distilled Large Language Models (LLMs) without requiring a teacher model at inference time. By identifying sparse "divergence points" and re-ranking the student's top candidate tokens using a lightweight adapter, SRA achieves significant gains in mathematical reasoning (up to 4.37% Pass@16) while keeping the base model frozen.

What Problem Does This Paper Address?

Knowledge distillation is the standard way to shrink massive LLMs into "student" models that are cheap to run. However, these smaller models often hit a performance ceiling, especially in complex, multi-step reasoning tasks like math or coding.

While "collaborative inference"—where a small student generates most tokens but asks a large teacher for help at difficult steps—can close this gap, it’s a deployment nightmare. It requires keeping a massive teacher model in memory and introduces significant latency. The authors ask a compelling question: Can we "internalize" these teacher corrections into a lightweight mechanism that operates independently?

Methods

The authors started with two key empirical observations:

  1. Sparsity: The student only diverges from the teacher at a small fraction of positions (roughly 14%).
  2. Candidate Coverage: Even when the student picks the wrong token, the teacher's preferred token is almost always sitting in the student's "top-K" list.

Based on this, they developed SRA, which consists of two modules added on top of a frozen student model:

  • Discrepancy Gate (DG): A tiny MLP that monitors the student's internal hidden states. It predicts whether the current token is a "divergence point" where the student is likely to stray from the teacher's logic.
  • Candidate Residual Adapter (CRA): If the gate triggers, this module (a small bidirectional Transformer) looks at the student's hidden states and the embeddings of the top-K candidate tokens. It predicts "logit residuals"—small adjustments to the scores—to re-rank those candidates and steer the student back on track.
Figure 2: Overview of SRA. The upper panel shows inference. DG reads the final hidden state ht of the frozen student and decides whether to correct the current step. If gϕ(st) = 1, CRA predicts logit residuals over the student’s top-K candidates and the next token is sampled from qθ, otherwise it is sampled from πS. The lower panel shows training, where the teacher provides qT and ℓ(st) to supervise CRA and DG in two stages.

Training happens in two stages: a "Cold Start" phase using offline student trajectories, followed by an "On-Policy" phase where the adapter is refined on trajectories generated by the newly enhanced policy to handle distribution shift.

Datasets

  • Training: The models were trained on DAPO-Math-17k, a dataset focused on mathematical reasoning.
  • Evaluation: The primary benchmarks were math-heavy: AIME24/25, AMC23, and MATH-500.
  • Generalization: To ensure the model didn't lose its general abilities, the authors also tested on GPQA-Diamond (science), HumanEval (coding), and MMLU-Pro (multidisciplinary).

Key Insights and Ideas

  • TV Distance > KL Divergence: The authors found that Total Variation (TV) distance is a much more sensitive indicator for training the Discrepancy Gate than standard KL divergence. TV distance grows linearly with small probability shifts, making it easier for a small MLP to detect when a top-1 token is about to flip.
  • Internalizing Collaboration: SRA essentially acts as a "cached" version of a teacher's wisdom. Instead of calling the teacher, the student uses the CRA to simulate what the teacher would have done at that specific difficult step.
  • Preserving Diversity: Because SRA only intervenes at selected positions, it avoids the "mode collapse" often seen in standard fine-tuning, preserving the model's ability to explore different solution paths (crucial for metrics like Pass@K).

Key Results

The researchers tested SRA on the Qwen3 family (0.6B to 4B parameters).

  • Consistent Gains: SRA improved the average Pass@16 across nearly all settings. For the Qwen3-4B model, it raised the average Pass@16 by 4.37 percentage points over the distilled baseline.
  • Beating the Teacher: In some cases, like AIME25, the SRA-enhanced 4B model actually surpassed the 8B teacher model it was learning from.
  • Overcoming Plateaus: The authors showed that SRA provides better gains than simply continuing standard distillation for more epochs, suggesting it captures nuances that global loss functions miss.
Main results table showing performance gains across different model sizes and benchmarks.

Limitations and Future Work

  • Top-K Assumption: SRA relies on the teacher's preferred token being in the student's top-K. While this holds true for structured reasoning, it might be less reliable in open-ended, creative generation where the vocabulary space is less constrained.
  • Training Overhead: While inference is very efficient, training requires on-policy rollouts and teacher forward passes, which is computationally expensive.
  • Scope: The current work focuses heavily on structured reasoning (math/code). The authors note that exploring how this applies to free-form dialogue is a key next step.

Summary updated

Abstract

Knowledge distillation transfers reasoning capabilities from large language models to lightweight students. However, distilled small models still struggle with multi-step reasoning. We observe that the distribution divergences between teacher and student models along reasoning paths are highly sparse, and target tokens almost always fall within the student top-$K$ candidate set. Based on this finding, unlike approaches updating student backbones or requiring runtime teacher computation, we propose Selective Residual Adaptation (SRA), a plug-and-play framework that internalizes localized corrections. Concretely, SRA employs a Discrepancy Gate to identify divergence positions alongside a Candidate Residual Adapter for top-$K$ logit calibration to correct the mismatch. Theoretically, we establish error bounds ensuring strict sequence level performance gains. Across mathematical benchmarks on Qwen3 models from 0.6B to 4B, SRA consistently enhances distilled student models, raising average Pass@16 by up to 4.37 percentage points while preserving general capabilities.
BibTeX
@misc{dou2026sra,
  author = {Zhenxing Dou and Kun He and Mingyang Yi and Yong Liu},
  title = {{SRA: Selective Residual Adaptation for LLM Reasoning Enhancement}},
  year = {2026},
  eprint = {2610.34199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34199},
}
Figure