TL;DR
Large language models (LLMs) struggle with complex reasoning due to sparse rewards. While on-policy distillation (OPD) helps, it treats all teacher signals equally, leading to issues with unreliable guidance and diversity loss. This paper introduces SCOPE, a novel framework that intelligently routes and weights training signals based on trajectory correctness and teacher/student confidence, significantly improving reasoning performance and diversity.
What Problem Does This Paper Address?
Training LLMs for complex reasoning tasks, like mathematical problem-solving, often relies on reinforcement learning (RL). A major hurdle in RL is the "credit assignment problem": when a model makes a mistake, it's hard to pinpoint which specific token or decision led to the error, especially when rewards are only given at the very end of a long sequence of generated tokens.
On-Policy Distillation (OPD) attempts to solve this by using a "teacher" LLM to provide denser, token-level guidance to the "student" LLM. This guidance comes in the form of KL divergence, encouraging the student to mimic the teacher's probability distribution over tokens. However, standard OPD has two main weaknesses:
- Unreliable Teacher Guidance: For incorrect trajectories (where the student fails), the teacher's guidance can be noisy if the teacher itself is unsure or if the student's flawed prefix confuses the teacher. OPD applies this potentially noisy guidance uniformly, hindering effective error correction.
- Diversity Degradation: For correct trajectories (where the student succeeds), uniform guidance can over-reinforce dominant reasoning paths, leading to "mode collapse" and a loss of diverse, potentially valid, but less common, reasoning strategies. This is particularly problematic for metrics like Pass@k, which measure performance across multiple generated solutions.
SCOPE aims to address these issues by introducing a more nuanced approach to on-policy distillation.
Methods
SCOPE introduces a dual-path adaptive training framework that intelligently routes and weights training signals based on the correctness of the student's generated trajectory and the confidence of the teacher/student models.
- Outcome-Driven Group Branching: First, SCOPE routes the student's generated trajectories into two distinct groups:
- Correct Trajectories ($\Omega_c$): Where the student successfully solves the problem.
- Incorrect Trajectories ($\Omega_w$): Where the student fails.
- Dual-Path Adaptive Weighting (DPAW):
- For Incorrect Trajectories ($\Omega_w$): SCOPE employs teacher-perplexity-weighted KL distillation. The idea is that if the teacher model has low perplexity (meaning it's confident) on a flawed prefix, its guidance is more likely to be accurate and helpful for correction. Conversely, high teacher perplexity indicates uncertainty or noise, so these trajectories are down-weighted. This filters out "prefix-induced noise."
- For Correct Trajectories ($\Omega_c$): SCOPE uses student-perplexity-weighted Maximum Likelihood Estimation (MLE). Here, the goal is to encourage diversity. Instead of uniformly reinforcing all correct answers, SCOPE up-weights correct trajectories where the student has higher perplexity (i.e., lower confidence or more unconventional paths). This helps preserve and explore low-probability but valid reasoning strategies, counteracting mode collapse.
- Group-Level Normalization: Both paths use a group-level normalization mechanism. This adaptively calibrates the weights within each group of trajectories for a given prompt, accounting for varying prompt difficulties and stabilizing the training process.
The overall SCOPE objective combines these weighted losses from both paths.
Datasets
The paper evaluates SCOPE on several mathematical reasoning benchmarks:
- AIME24, AIME25
- AMC23
- MATH500
- Minerva
- OlympiadBench
Additionally, extended experiments demonstrate SCOPE's applicability to code generation tasks using:
- TACO dataset (for training)
- LiveCodeBench, HumanEval, and Codeforces (for evaluation)
Key Insights and Ideas
- Signal Quality Matters: The core insight is that not all teacher signals are created equal. The reliability of a teacher's guidance depends heavily on the context (the student's prefix) and the teacher's own confidence (indicated by perplexity).
- Perplexity as a Proxy: Teacher perplexity is a good indicator of the teacher's ability to provide reliable corrective guidance on incorrect trajectories. Student perplexity can indicate exploration of less common but valid reasoning paths on correct trajectories.
- The Pass@k Paradox: Uniformly reinforcing correct trajectories can lead to a decline in performance on metrics like Pass@k (where k > 1) because it amplifies dominant reasoning modes at the expense of diversity. SCOPE directly tackles this.
- The Flawed Prefix Trap: When a student's prefix is flawed, a teacher's guidance can become noisy and uninformative, leading to inefficient or unstable learning. SCOPE's teacher-guided weighting actively filters out this noise.
- Dual-Path is Key: Separating correct and incorrect trajectories and applying different weighting strategies to each path is crucial for simultaneously improving error correction and preserving reasoning diversity.
Key Results
- Mathematical Reasoning: SCOPE achieves significant improvements over baselines like GRPO and standard OPD.
- Average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 across six mathematical reasoning benchmarks.
- Outperforms standard OPD by +5.54% in Avg@32 and +2.60% in Pass@32 on average.
- Demonstrates strong gains on specific benchmarks like Olympiad (+10.69% Avg@32) and AMC23 (+6.59% Avg@32).
- Pass@k Performance: SCOPE shows superior diversity scaling, with Pass@k rates continuing to improve for larger k, unlike baselines that plateau due to mode collapse.
- Code Generation: SCOPE also shows broad applicability, improving Avg@32 by +4.69% on average across HumanEval, LiveCodeBench, and Codeforces, and outperforming GRPO by +4.21%.
- Ablation Studies: Removing or reversing the adaptive weighting mechanisms (student-guided or teacher-guided) leads to significant performance degradation, highlighting the complementary nature of the two paths.
Limitations and Future Work
- Dependence on Verifiable Outcomes: SCOPE relies on automatically verifiable outcome signals (like correct/incorrect answers) to split trajectories. This limits its direct application to domains where such signals are not readily available (e.g., open-ended dialogue, creative writing) without additional reward modeling or verification mechanisms.
- Computational Resources: The empirical evaluation was constrained by computational resources, suggesting that further experiments on larger foundation models, Mixture-of-Experts (MoE) architectures, and more diverse verifiable domains would be beneficial.
- Future Work: The authors suggest exploring more sophisticated reward models for subjective tasks and scaling SCOPE to larger and more complex LLM architectures.
Summary updated