TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) often suffers from entropy collapse and poor exploration because popular algorithms like GRPO apply a single, uniform advantage score to every token in a generated sequence. UCAS (UnCertainty-aware Advantage Shaping) fixes this without requiring an external process reward model by reshaping advantages at two levels: boosting rewards for correct-yet-uncertain sequences while penalizing overconfident errors, and applying a per-token penalty based on raw logits to deter local overconfidence.
What Problem Does This Paper Address?
Reinforcement learning methods using binary or rule-based checks—such as verifying whether a math answer is correct—have become the engine behind reasoning models like DeepSeek-R1. Typically, algorithms such as Group Relative Policy Optimization (GRPO) compute an outcome-based advantage across a group of rollouts and assign that exact same scalar reward to every generated token.
This creates a coarse credit assignment problem:
- Trivial filler tokens and high-stakes reasoning steps are rewarded or penalized identically.
- The policy rapidly converges to narrow, "safe" paths, causing entropy collapse (a severe loss of output diversity).
- While the model might achieve decent greedy accuracy (pass@1), its ability to explore diverse solutions under repeated sampling (pass@@k) severely degrades.
Training step-by-step process reward models (PRMs) is expensive, noisy, and prone to reward hacking. The authors ask: Can we achieve fine-grained, step-sensitive credit assignment using only the policy model's own internal uncertainty signals?
Methods
UCAS is an extra-model-free, two-stage advantage shaping framework integrated into RLVR:
- Stage 1: Response-Level Confidence Modulation (Macro-level)
- The overall trajectory confidence $C(o_i|q)$ is computed as the average KL-divergence between a uniform vocabulary distribution and the model's next-token predictive distributions.
- Confidences across sampled candidate rollouts are standardized via $z$-scores ($\hat{C}_i$).
- The sequence advantage $\hat{A}_i$ is scaled exponentially: $$\hat{A}_i^{\text{mod}} = \begin{cases} \exp(-\alpha \cdot \hat{C}_i) \cdot \hat{A}_i & \text{if } \hat{A}_i > 0 \text{ (correct)} \\ \exp(\alpha \cdot \hat{C}_i) \cdot \hat{A}_i & \text{if } \hat{A}_i < 0 \text{ (incorrect)} \end{cases}$$
- Intuition: Reward exploration by giving an extra bonus to correct answers discovered via uncertain reasoning paths, while harshly penalizing confident reasoning trajectories that end in incorrect answers.
- Stage 2: Token-Level Certainty Penalty (Micro-level)
- Because post-softmax probabilities can be poorly calibrated, UCAS uses the unnormalized raw logit $\ell_{i,t}$ of the chosen token as a proxy for local certainty.
- Logits are min-max normalized across the sequence to yield $\hat{\ell}_{i,t} \in [0, 1]$.
- A regularizing penalty is subtracted from the advantage: $$\hat{A}_{i,t}^{\text{UCAS}} = \hat{A}_i^{\text{mod}} - \beta \cdot \hat{\ell}_{i,t}$$
- Intuition: Enforces "epistemic humility" at individual generation steps to keep the output distribution from collapsing into a delta function.
Datasets
- Training: MATH dataset (specifically challenging problems from levels 3 to 5).
- In-Domain Evaluation (Math): Five standard benchmarks totaling 1,560 problems: AIME24, MATH-500, AMC, Minerva, and OlympiadBench.
- Cross-Domain Generalization: LeetCode (code generation), LiveCodeBench (competitive programming), and MMLU (multitask reasoning).
- No new benchmarks were introduced.
Key Insights and Ideas
- Entropy Recovery vs. Collapse: While standard GRPO displays a monotonic decrease in generation entropy throughout training, UCAS exhibits an initial dip followed by a steady rebound and stabilization at a higher entropy level, all while test rewards keep climbing.
- Logits Over Post-Softmax Probabilities: Raw logits serve as a more sensitive, uncompressed indicator of token certainty than post-softmax probabilities, which tend to mask token-level uncertainty differences due to calibration saturation.
- Exploration Incentive Over Conservative Gating: Prior uncertainty-aware methods (like SEED-GRPO) attenuate updates when uncertainty is high to avoid risky steps. UCAS flips this logic: it deliberately amplifies the learning signal for correct paths discovered under uncertainty, treating high uncertainty as an exploration bonus.
- Longer, Deeper Reasoning Chains: Models trained with UCAS autonomously develop longer chains of thought during training, reflecting more thorough, self-correcting problem exploration.
Key Results
Experiments were conducted using Qwen2.5-Math (1.5B and 7B scales):
- Benchmark Performance (Greedy Pass@1):
- 1.5B Model: UCAS achieved an average of 47.3% across all five math benchmarks, outperforming DAPO (41.2%), GRPO (40.4%), and SEED-GRPO (43.5%).
- 7B Model: UCAS reached an average of 56.7%, surpassing DAPO (50.5%), GRPO (48.9%), KTAE (52.6%), and SEED-GRPO (54.7%), with notable gains on AIME24 (43.3% vs. 30.5% for DAPO).
- Pass@$k$ Scaling: On AIME24, UCAS scales much more effectively with rollout count $k$ than vanilla GRPO, demonstrating sustained diversity rather than premature convergence.
- Cross-Domain Transfer: Even though it was trained solely on math problems, the 7B UCAS model improved performance over DAPO on code and general reasoning:
- LeetCode: 23.6% vs. 18.3% (+5.3)
- LiveCodeBench: 14.8% vs. 9.2% (+5.6)
- MMLU: 70.8% vs. 67.3% (+3.5)
Limitations and Future Work
- Strictly Verifiable Tasks: The study is built on tasks with deterministic, rule-checkable ground truths (e.g., math answers). Adapting UCAS to subjective domains (creative writing, dialogue, summarization) where rewards are dense or noisy remains untested.
- Uncertainty Estimator Choice: The approach currently relies on vocabulary-wide KL divergence and token logits. The authors note that exploring richer uncertainty measures—such as Monte Carlo dropout, model ensembles, or semantic clustering—could further refine the method.
- Hyperparameter Sensitivity: Aggressive sequence modulation ($\alpha \ge 0.4$) causes "signal dominance," where uncertainty scaling overshadows actual correctness signals and degrades performance.
Summary updated