SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision

Chia-Hsuan Lee, Zelei Cheng, Yu Wang, Renkun Ni, Sambit Sahu, Shi-Xiong Zhang, William M. Campbell

TL;DR

SEAD (Sparse Entropy-Adaptive Distillation) improves how we train small reasoning models by recognizing that teacher feedback is only useful when a student is "competent" enough to understand it. By using entropy to filter out redundant tokens, scheduling problem difficulty, and annealing training objectives, SEAD achieves a +4.8% accuracy boost on math benchmarks while skipping 50% of unnecessary computation.

What Problem Does This Paper Address?

To make smaller, deployable AI models as smart as their massive counterparts, researchers use Knowledge Distillation (KD). The gold standard is On-Policy Distillation (OPD), where a student model generates its own reasoning "rollouts" and a larger teacher model corrects them.

However, the authors identify a major flaw in current OPD: supervision quality depends on student competence. 1. The Noise Problem: If a student is totally lost on a hard math problem, its output is gibberish. Per-token corrections from a teacher on gibberish are just "noise" that confuses the student. 2. The Waste Problem: If a student has already mastered a concept (like basic arithmetic or formatting), getting teacher feedback on those tokens is a waste of compute. 3. The Uniformity Trap: Existing methods treat every token, every training phase, and every prompt with the same "one-size-fits-all" approach.

Methods

The authors introduce SEAD (Sparse Entropy-Adaptive Distillation), which uses entropy (a measure of uncertainty) as a "unified probe" to fix these issues at three different scales:

  • Token-Level (The Zones): SEAD looks at the joint entropy of the teacher and student to partition tokens into three zones:
  • Zone A (Skip): Both models are confident. These are redundant tokens (like "Therefore," or "1+1=2") and are skipped to save compute (~50% of tokens).
  • Zone B (Refine): Teacher is confident, student is not. SEAD uses Reverse KL divergence to "sharpen" the student's focus on the teacher's correct path.
  • Zone C (Explore): Both are uncertain (e.g., at a complex reasoning fork). SEAD uses Forward KL to preserve diversity and prevent the student from "collapsing" into a single narrow way of thinking.
  • Temporal-Level (Annealing): A continuous cosine schedule transitions the training from exploration-heavy (Forward KL) to refinement-heavy (Reverse KL) as the student gets smarter.
  • Prompt-Level (Curriculum): A competence-gated curriculum introduces math problems from easy to hard. This ensures the student's rollouts are coherent enough for the teacher's feedback to actually make sense.

Datasets

The models were trained on a deduplicated version of the DAPO-Math-17K dataset. Evaluation was performed across six rigorous mathematical reasoning benchmarks:

  • Standard Reasoning: MATH-500, Minerva-Math, and OlympiadBench.
  • Competition-Level: AMC 2023, AIME 2024, and AIME 2025 (where reasoning depth is most critical).

Key Insights and Ideas

  • Symbiotic Necessity: The most striking finding is that these three components (Zones, Annealing, Curriculum) are not just independent tricks—they are a "virtuous cycle." For example, the token-level zones only work if the curriculum provides coherent rollouts to analyze.
  • Super-Additive Gains: Combining Token Zones with Annealing yielded a 20x amplification in performance compared to using either one in isolation.
  • Efficiency is Free: You can discard 50% of the tokens (Zone A) while retaining over 90% of the training signal. This proves that "dense" supervision is largely redundant in reasoning tasks.
  • Preventing Entropy Collapse: Vanilla OPD often suffers from "entropy collapse," where the student becomes prematurely overconfident and loses the ability to explore different problem-solving strategies. SEAD’s adaptive approach keeps the student's "mind" open longer.

Key Results

The authors tested SEAD on two model families: OLMo-3 (7B student, 32B teacher) and Nemotron (8B student, 49B teacher).

  • Overall Boost: SEAD achieved a +4.8% average accuracy increase over vanilla OPD.
  • Hard Math Mastery: The gains were most dramatic on the hardest benchmarks, with a +7.7% jump on AIME 2024 and +7.9% on AIME 2025.
  • Generalization: Even when the gap between the teacher and student was small (only 2.2 points in the Nemotron case), SEAD still provided meaningful improvements where standard baselines stagnated.

Limitations and Future Work

  • Static Curriculum: The current difficulty scores are calculated before training starts. The authors suggest that "adaptive re-estimation" (updating difficulty on the fly) could be even more efficient.
  • Domain Specificity: The study focused entirely on mathematics. Whether these entropy-guided zones translate well to coding or creative writing remains an open question.
  • Long-Chain Reasoning: The behavior of SEAD on extremely long reasoning chains (over 8,000 tokens) hasn't been fully explored yet.

Summary updated

Abstract

On-policy distillation (OPD) has a property absent in offline distillation and RL: teacher supervision quality depends on student competence. Incoherent rollouts yield noisy gradients; already-mastered tokens yield redundant ones. This creates waste at three scales—tokens, training phases, and prompts—yet existing methods supervise uniformly. We introduce SEAD, which uses entropy as a unified probe of this competence-dependent degradation at three scales: (1) joint teacher-student entropy partitions tokens into zones receiving tailored divergences or zero gradient (~50% skipped); (2) a cosine schedule anneals from forward to reverse KL as competence grows; (3) a competence-gated curriculum introduces prompts easy-to-hard. These components are symbiotically necessary: token selection requires coherent rollouts (curriculum), and annealing requires monotonic improvement (also curriculum). On OLMo-3 (7B to 32B), SEAD achieves +4.8 average accuracy over vanilla OPD across six math benchmarks, with ablations confirming super-additive interactions.
BibTeX
@inproceedings{lee2026sead,
  author = {Chia-Hsuan Lee and Zelei Cheng and Yu Wang and Renkun Ni and Sambit Sahu and Shi-Xiong Zhang and William M. Campbell},
  title = {{SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision}},
  booktitle = {AI4Math @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=ujbngu2N8p},
}
Figure