Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning

Can Xie, Ruotong Pan, Xiangyu Wu, Zhang yunfei, Jiayi Fu, Tingting Gao, Guorui ZhouSubmission to ICLR 2026 · Withdrawn

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) often suffers from entropy collapse and poor exploration because popular algorithms like GRPO apply a single, uniform advantage score to every token in a generated sequence. UCAS (UnCertainty-aware Advantage Shaping) fixes this without requiring an external process reward model by reshaping advantages at two levels: boosting rewards for correct-yet-uncertain sequences while penalizing overconfident errors, and applying a per-token penalty based on raw logits to deter local overconfidence.

What Problem Does This Paper Address?

Reinforcement learning methods using binary or rule-based checks—such as verifying whether a math answer is correct—have become the engine behind reasoning models like DeepSeek-R1. Typically, algorithms such as Group Relative Policy Optimization (GRPO) compute an outcome-based advantage across a group of rollouts and assign that exact same scalar reward to every generated token.

This creates a coarse credit assignment problem:

  • Trivial filler tokens and high-stakes reasoning steps are rewarded or penalized identically.
  • The policy rapidly converges to narrow, "safe" paths, causing entropy collapse (a severe loss of output diversity).
  • While the model might achieve decent greedy accuracy (pass@1), its ability to explore diverse solutions under repeated sampling (pass@@k) severely degrades.

Training step-by-step process reward models (PRMs) is expensive, noisy, and prone to reward hacking. The authors ask: Can we achieve fine-grained, step-sensitive credit assignment using only the policy model's own internal uncertainty signals?

Methods

UCAS is an extra-model-free, two-stage advantage shaping framework integrated into RLVR:

  1. Stage 1: Response-Level Confidence Modulation (Macro-level)
  2. The overall trajectory confidence $C(o_i|q)$ is computed as the average KL-divergence between a uniform vocabulary distribution and the model's next-token predictive distributions.
  3. Confidences across sampled candidate rollouts are standardized via $z$-scores ($\hat{C}_i$).
  4. The sequence advantage $\hat{A}_i$ is scaled exponentially: $$\hat{A}_i^{\text{mod}} = \begin{cases} \exp(-\alpha \cdot \hat{C}_i) \cdot \hat{A}_i & \text{if } \hat{A}_i > 0 \text{ (correct)} \\ \exp(\alpha \cdot \hat{C}_i) \cdot \hat{A}_i & \text{if } \hat{A}_i < 0 \text{ (incorrect)} \end{cases}$$
  5. Intuition: Reward exploration by giving an extra bonus to correct answers discovered via uncertain reasoning paths, while harshly penalizing confident reasoning trajectories that end in incorrect answers.
  6. Stage 2: Token-Level Certainty Penalty (Micro-level)
  7. Because post-softmax probabilities can be poorly calibrated, UCAS uses the unnormalized raw logit $\ell_{i,t}$ of the chosen token as a proxy for local certainty.
  8. Logits are min-max normalized across the sequence to yield $\hat{\ell}_{i,t} \in [0, 1]$.
  9. A regularizing penalty is subtracted from the advantage: $$\hat{A}_{i,t}^{\text{UCAS}} = \hat{A}_i^{\text{mod}} - \beta \cdot \hat{\ell}_{i,t}$$
  10. Intuition: Enforces "epistemic humility" at individual generation steps to keep the output distribution from collapsing into a delta function.

Datasets

  • Training: MATH dataset (specifically challenging problems from levels 3 to 5).
  • In-Domain Evaluation (Math): Five standard benchmarks totaling 1,560 problems: AIME24, MATH-500, AMC, Minerva, and OlympiadBench.
  • Cross-Domain Generalization: LeetCode (code generation), LiveCodeBench (competitive programming), and MMLU (multitask reasoning).
  • No new benchmarks were introduced.

Key Insights and Ideas

  • Entropy Recovery vs. Collapse: While standard GRPO displays a monotonic decrease in generation entropy throughout training, UCAS exhibits an initial dip followed by a steady rebound and stabilization at a higher entropy level, all while test rewards keep climbing.
  • Logits Over Post-Softmax Probabilities: Raw logits serve as a more sensitive, uncompressed indicator of token certainty than post-softmax probabilities, which tend to mask token-level uncertainty differences due to calibration saturation.
  • Exploration Incentive Over Conservative Gating: Prior uncertainty-aware methods (like SEED-GRPO) attenuate updates when uncertainty is high to avoid risky steps. UCAS flips this logic: it deliberately amplifies the learning signal for correct paths discovered under uncertainty, treating high uncertainty as an exploration bonus.
  • Longer, Deeper Reasoning Chains: Models trained with UCAS autonomously develop longer chains of thought during training, reflecting more thorough, self-correcting problem exploration.

Key Results

Experiments were conducted using Qwen2.5-Math (1.5B and 7B scales):

  • Benchmark Performance (Greedy Pass@1):
  • 1.5B Model: UCAS achieved an average of 47.3% across all five math benchmarks, outperforming DAPO (41.2%), GRPO (40.4%), and SEED-GRPO (43.5%).
  • 7B Model: UCAS reached an average of 56.7%, surpassing DAPO (50.5%), GRPO (48.9%), KTAE (52.6%), and SEED-GRPO (54.7%), with notable gains on AIME24 (43.3% vs. 30.5% for DAPO).
  • Pass@$k$ Scaling: On AIME24, UCAS scales much more effectively with rollout count $k$ than vanilla GRPO, demonstrating sustained diversity rather than premature convergence.
  • Cross-Domain Transfer: Even though it was trained solely on math problems, the 7B UCAS model improved performance over DAPO on code and general reasoning:
  • LeetCode: 23.6% vs. 18.3% (+5.3)
  • LiveCodeBench: 14.8% vs. 9.2% (+5.6)
  • MMLU: 70.8% vs. 67.3% (+3.5)

Limitations and Future Work

  • Strictly Verifiable Tasks: The study is built on tasks with deterministic, rule-checkable ground truths (e.g., math answers). Adapting UCAS to subjective domains (creative writing, dialogue, summarization) where rewards are dense or noisy remains untested.
  • Uncertainty Estimator Choice: The approach currently relies on vocabulary-wide KL divergence and token logits. The authors note that exploring richer uncertainty measures—such as Monte Carlo dropout, model ensembles, or semantic clustering—could further refine the method.
  • Hyperparameter Sensitivity: Aggressive sequence modulation ($\alpha \ge 0.4$) causes "signal dominance," where uncertainty scaling overshadows actual correctness signals and degrades performance.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). However, prevailing algorithms like GRPO broadcast a uniform advantage signal across all tokens in a sequence. This coarse-grained approach overlooks the pivotal role of uncertain, high-stakes decisions during reasoning, leading to inefficient exploration and the well-documented problem of entropy collapse. To address this, we introduce $\textbf{U}$n$\textbf{C}$ertainty-aware $\textbf{A}$dvantage $\textbf{S}$haping ($\textbf{UCAS}$), a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. UCAS operates in two stages: it first modulates the response-level advantage using the model's overall self-confidence, and then applies a token-level penalty based on raw logit certainty. This dual mechanism encourages exploration of high-uncertainty paths that yield correct answers while penalizing overconfident yet erroneous reasoning, effectively balancing the exploration-exploitation trade-off. Extensive experiments on five mathematical reasoning benchmarks show that UCAS significantly outperforms strong RLVR baselines across multiple model scales, including 1.5B and 7B. Our analysis confirms that UCAS not only achieves higher rewards but also promotes greater reasoning diversity and successfully mitigates entropy collapse.
BibTeX
@misc{xie2025unlocking,
  author = {Can Xie and Ruotong Pan and Xiangyu Wu and Zhang yunfei and Jiayi Fu and Tingting Gao and Guorui Zhou},
  title = {{Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning}},
  year = {2025},
  eprint = {2510.10649},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10649},
}
Figure