Introduction: The Pass@1 vs. Pass@k Dilemma
Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone technique for enhancing the reasoning capabilities of Large Language Models (LLMs) on tasks with objective answers, such as mathematics and coding. While RLVR reliably improves a model's ability to generate the correct answer on its first try ($pass@1$), it often fails to yield comparable gains in $pass@k$—the model's ability to find the correct answer when allowed multiple attempts.
This discrepancy raises a fundamental question: Does RLVR genuinely teach LLMs novel reasoning abilities, or does it merely train them to more efficiently sample the reasoning paths they already know? Prior analyses suggest the latter, pointing to a phenomenon where models "collapse" onto a few familiar reasoning paths, failing to discover diverse solution strategies.
The "Invisible Leash" Paradox
The culprit behind this mode collapse is a structural constraint in standard RLVR objectives: reverse-KL divergence regularization.
During reinforcement learning, the KL divergence term acts as a mathematical "anchor" or leash. It prevents the model from drifting too far from its original base distribution, which is crucial for preventing "reward hacking"—a scenario where the model discovers nonsensical shortcuts to achieve a high reward score. However, this anchor inherently forces the policy to stick to pre-existing reasoning patterns, suppressing the emergence of alternative, potentially superior reasoning modes.
Previous attempts to solve this issue have proven inadequate. Removing the KL leash entirely leads to unstable training and reward hacking. Conversely, replacing it with forward-KL divergence disrupts the efficiency-coverage trade-off, causing the model to allocate probability mass to irrelevant, "off-target" regions.
SAGE: Moving the Post, Not Cutting the Leash
To resolve this tension, researchers introduce SAGE (Shaping Anchors for Guided Exploration). SAGE is a principled framework that enables controllable exploration without sacrificing the stability provided by KL regularization.
Instead of weakening or removing the anchor, SAGE reshapes the reference distribution itself. It introduces a multiplicative guide function, $q(x, y)$, which modifies the reference distribution. In essence, SAGE anchors the model not to its exact original self, but to a version of itself that is gently "pushed" toward more interesting, underexplored reasoning areas. By utilizing a "pseudo-KL" objective, SAGE implements this reshaping efficiently without requiring the guide function to be a perfectly normalized probability distribution.
Entropy as a Reasoning Compass
The authors propose three distinct methods for constructing the guide function:
- Random Exploration: Adding stochastic noise as a baseline.
- Token-Level Exploration: Utilizing surprisal (token rarity) to encourage unique phrasing and vocabulary.
- Branch-Level Exploration: Utilizing entropy to identify logical "decision points."
The Branch-Level approach proved to be the most effective. High entropy serves as an excellent signal for a "reasoning branch"—a juncture in a problem where multiple logical paths are viable. When the model is uncertain about the next step (high entropy), SAGE amplifies exploration specifically at that branch, encouraging the model to investigate alternative logical routes.
Experimental Results and Key Insights
The researchers evaluated SAGE across rigorous mathematical reasoning benchmarks, including AIME (2024-2025), AMC23, and MATH-500. The primary backbone model was Qwen-2.5-Math-7B, with additional tests on DeepSeek-R1-Distill-Qwen-7B to validate efficacy on long-reasoning models.
Consistent Gains in Accuracy and Diversity
Standard RLVR typically forces a trade-off: as $pass@1$ (accuracy) increases, $pass@k$ (diversity) drops due to mode collapse. SAGE breaks this paradigm, improving both simultaneously. On the highly challenging AIME benchmark, the Branch-level SAGE variant consistently outperformed standard GRPO and other baselines, improving $pass@1$ from 5.3% to 6.3% while significantly boosting $pass@256$.
Mode Discovery vs. Mode Sharpening
While standard RLVR merely "sharpens" existing knowledge, SAGE enables genuine "mode discovery." Qualitative case studies revealed a shift toward more rigorous reasoning. Base models often relied on "guess-and-check" heuristics, whereas SAGE-trained models learned to employ systematic constraint verification and alternative logical approaches, such as proof by contradiction. Furthermore, SAGE demonstrated superior robustness against mode collapse on out-of-distribution (OOD) tasks like the Knights and Knaves logic puzzles.
Limitations and Future Directions
Despite its successes, SAGE operates under a "reference support bound." Because it redistributes probability among paths the original model could potentially take, it cannot discover a reasoning strategy from scratch if that strategy has exactly zero probability in the base model. Additionally, while entropy and surprisal are effective heuristic signals, they remain relatively simple intrinsic metrics.
Future work may explore using stronger "teacher" models to design more expressive guide functions. By continuing to refine how we shape the anchor, frameworks like SAGE pave the way for LLMs that don't just memorize the most common path to an answer, but genuinely explore the vast landscape of logical reasoning.
Summary updated