TL;DR
Reinforcement learning dramatically boosts LLM reasoning performance, but it often leaves models dangerously overconfident when they are wrong. This paper introduces Reinforcement Learning with Confidence Margin (RLCM), a framework that uses process supervision to widen the confidence gap between correct and incorrect intermediate reasoning steps. The result is a model that maintains state-of-the-art reasoning accuracy while delivering highly calibrated, trustworthy confidence scores.
What Problem Does This Paper Address?
Scaling test-time compute via Reinforcement Learning (RL)—the paradigm behind models like DeepSeek-R1 and OpenAI's o1—has unlocked massive performance gains in complex mathematics, coding, and science. However, these models suffer from a hidden flaw: calibration degeneration.
Because standard RL algorithms reward models solely based on whether the final answer is right or wrong, they incentivize models to be aggressively overconfident. This leads to hallucinations, makes it difficult to know when to trust a model's output, and wastes valuable compute on unsolvable problems. Previous attempts to fix this by directly penalizing miscalibration (e.g., using Brier score rewards) often tank the model's reasoning accuracy, creating a harsh trade-off between being smart and being honest.
Methods
The authors propose RLCM (Reinforcement Learning with Confidence Margin). Instead of forcing absolute confidence scores to match empirical accuracy at every single token, RLCM treats calibration as a ranking problem across intermediate states within a single reasoning trajectory.
As shown in the overview above, the method operates via four key steps:
- Intermediate Budgets: Predefined truncation points ($b_1, b_2, \dots, b_n$) are selected along a sampled reasoning path.
- Lightweight Probing: A simple 2-layer MLP probe is attached to the model's hidden states at these points to predict confidence ($C_b$). Crucially, gradients from this probe do not backpropagate into the core LLM, keeping training stable.
- Monte Carlo Estimation: To find the ground-truth "solvability" ($Y_b$) of an intermediate step, the model is forced to immediately output an answer from that step using $K$ parallel rollouts.
- Confidence Margin Reward: The intermediate steps are partitioned into "promising" steps (accuracy $\ge 0.5$) and "unpromising" steps (accuracy $< 0.5$). The model receives a process reward ($R_{\text{margin}}$) for maximizing the gap between the average confidence of these two sets.
This margin reward is combined with the standard final answer correctness reward ($R_{\text{ans}}$) to optimize the policy jointly.
Datasets
- Training: All models were trained on the math-centric GRPO-LEAD dataset using an R1-distilled Qwen-7B base model.
- In-Domain Evaluation: A suite of challenging math benchmarks including MATH-500, AMC, OlympiadBench, AIME 2024, and AIME 2025.
- Out-of-Domain Evaluation: To test generalization under severe distribution shifts, the authors evaluated on GPQA Diamond (graduate-level science), LogiQA (logical reasoning), and LiveCodeBench (coding).
Key Insights and Ideas
- Ranking over Absolute Matching: Forcing a model to hit exact, absolute confidence targets disrupts reasoning optimization. Encouraging a relative margin (simply ranking good steps higher than bad steps) provides a softer, more stable constraint that leaves the policy free to maximize correctness.
- The Bottleneck of Verbalized or Logit Confidence: Traditional calibration relies on models verbalizing their confidence (e.g., "I am 80% sure") or averaging token logits. The authors show verbalized confidence is too quantized (discrete), while logit averages get compressed into a narrow high-confidence band as chains of thought grow longer. Probing internal hidden states avoids both bottlenecks.
- Downstream Utility: Calibrated confidence directly enables efficient Conformal Risk Control (knowing exactly when to early-exit a long reasoning chain to save tokens) and Confidence-Weighted Aggregation (better voting mechanisms for parallel attempts).
Key Results
- The Best Pareto Frontier: RLCM achieves the strongest trade-off between accuracy and Expected Calibration Error (ECE) on difficult benchmarks like AIME24.
- Drastic ECE Reduction: On AIME24, RLCM maintains a high accuracy of 49.2% while dropping ECE to 0.061 (compared to GRPO's ECE of 0.108 and $C^2$GSPG's 0.334).
- Out-of-Domain Generalization: Despite being trained entirely on math, RLCM generalized robustly. On GPQA Diamond, it achieved the highest accuracy (47.9%) while keeping ECE low (0.102) compared to baseline inflation.
- Token Savings: In early-exit scenarios using Conformal Risk Control, RLCM's smooth confidence signals allowed it to track perfect calibration closely, saving significant compute tokens compared to standard GRPO (see Figure 6 above).
Limitations and Future Work
- Training Overhead: Estimating intermediate correctness requires running multiple Monte Carlo rollouts ($K=4$) during the training phase, which increases the computational cost of the RL training loop.
- Predefined Budgets: The framework relies on fixed token intervals (e.g., every 500 tokens) to evaluate intermediate states. The authors suggest that future work could explore dynamically identifying critical "decision points" or semantic milestones in the reasoning trace rather than relying on fixed compute budgets.
Summary updated