Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme, Anqi Liu

TL;DR

Reinforcement learning dramatically boosts LLM reasoning performance, but it often leaves models dangerously overconfident when they are wrong. This paper introduces Reinforcement Learning with Confidence Margin (RLCM), a framework that uses process supervision to widen the confidence gap between correct and incorrect intermediate reasoning steps. The result is a model that maintains state-of-the-art reasoning accuracy while delivering highly calibrated, trustworthy confidence scores.

What Problem Does This Paper Address?

Scaling test-time compute via Reinforcement Learning (RL)—the paradigm behind models like DeepSeek-R1 and OpenAI's o1—has unlocked massive performance gains in complex mathematics, coding, and science. However, these models suffer from a hidden flaw: calibration degeneration.

Because standard RL algorithms reward models solely based on whether the final answer is right or wrong, they incentivize models to be aggressively overconfident. This leads to hallucinations, makes it difficult to know when to trust a model's output, and wastes valuable compute on unsolvable problems. Previous attempts to fix this by directly penalizing miscalibration (e.g., using Brier score rewards) often tank the model's reasoning accuracy, creating a harsh trade-off between being smart and being honest.

Methods

The authors propose RLCM (Reinforcement Learning with Confidence Margin). Instead of forcing absolute confidence scores to match empirical accuracy at every single token, RLCM treats calibration as a ranking problem across intermediate states within a single reasoning trajectory.

Figure 2: Overview of RLCM. We select intermediate compute budgets b1, . . . , bn along a reasoning trajectory. At each budget, Monte Carlo sampling estimates intermediate correctness, while a lightweight probe predicts confidence from the hidden state. Instead of directly aligning confidence with correctness, RLCM encourages a larger confidence margin between correct and incorrect intermediate states within the same trajectory.

As shown in the overview above, the method operates via four key steps:

  1. Intermediate Budgets: Predefined truncation points ($b_1, b_2, \dots, b_n$) are selected along a sampled reasoning path.
  2. Lightweight Probing: A simple 2-layer MLP probe is attached to the model's hidden states at these points to predict confidence ($C_b$). Crucially, gradients from this probe do not backpropagate into the core LLM, keeping training stable.
  3. Monte Carlo Estimation: To find the ground-truth "solvability" ($Y_b$) of an intermediate step, the model is forced to immediately output an answer from that step using $K$ parallel rollouts.
  4. Confidence Margin Reward: The intermediate steps are partitioned into "promising" steps (accuracy $\ge 0.5$) and "unpromising" steps (accuracy $< 0.5$). The model receives a process reward ($R_{\text{margin}}$) for maximizing the gap between the average confidence of these two sets.

This margin reward is combined with the standard final answer correctness reward ($R_{\text{ans}}$) to optimize the policy jointly.

Datasets

  • Training: All models were trained on the math-centric GRPO-LEAD dataset using an R1-distilled Qwen-7B base model.
  • In-Domain Evaluation: A suite of challenging math benchmarks including MATH-500, AMC, OlympiadBench, AIME 2024, and AIME 2025.
  • Out-of-Domain Evaluation: To test generalization under severe distribution shifts, the authors evaluated on GPQA Diamond (graduate-level science), LogiQA (logical reasoning), and LiveCodeBench (coding).

Key Insights and Ideas

  • Ranking over Absolute Matching: Forcing a model to hit exact, absolute confidence targets disrupts reasoning optimization. Encouraging a relative margin (simply ranking good steps higher than bad steps) provides a softer, more stable constraint that leaves the policy free to maximize correctness.
  • The Bottleneck of Verbalized or Logit Confidence: Traditional calibration relies on models verbalizing their confidence (e.g., "I am 80% sure") or averaging token logits. The authors show verbalized confidence is too quantized (discrete), while logit averages get compressed into a narrow high-confidence band as chains of thought grow longer. Probing internal hidden states avoids both bottlenecks.
  • Downstream Utility: Calibrated confidence directly enables efficient Conformal Risk Control (knowing exactly when to early-exit a long reasoning chain to save tokens) and Confidence-Weighted Aggregation (better voting mechanisms for parallel attempts).
Risk control and token savings with LTT

Key Results

  • The Best Pareto Frontier: RLCM achieves the strongest trade-off between accuracy and Expected Calibration Error (ECE) on difficult benchmarks like AIME24.
  • Drastic ECE Reduction: On AIME24, RLCM maintains a high accuracy of 49.2% while dropping ECE to 0.061 (compared to GRPO's ECE of 0.108 and $C^2$GSPG's 0.334).
Figure 4: Average accuracy (left) and ECE (right) over five in-domain datasets across token budgets, with correctness and confidence extracted via force output. RLCM matches the strongest baseline in accuracy while consistently yielding the lowest ECE.
  • Out-of-Domain Generalization: Despite being trained entirely on math, RLCM generalized robustly. On GPQA Diamond, it achieved the highest accuracy (47.9%) while keeping ECE low (0.102) compared to baseline inflation.
  • Token Savings: In early-exit scenarios using Conformal Risk Control, RLCM's smooth confidence signals allowed it to track perfect calibration closely, saving significant compute tokens compared to standard GRPO (see Figure 6 above).

Limitations and Future Work

  • Training Overhead: Estimating intermediate correctness requires running multiple Monte Carlo rollouts ($K=4$) during the training phase, which increases the computational cost of the RL training loop.
  • Predefined Budgets: The framework relies on fixed token intervals (e.g., every 500 tokens) to evaluate intermediate states. The authors suggest that future work could explore dynamically identifying critical "decision points" or semantic milestones in the reasoning trace rather than relying on fixed compute budgets.

Summary updated

Abstract

Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages the model to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation
BibTeX
@inproceedings{wang2026process,
  author = {Liaoyaqi Wang and Chunsheng Zuo and William Jurayj and Benjamin Van Durme and Anqi Liu},
  title = {{Process Supervision of Confidence Margin for Calibrated LLM Reasoning}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=8f8CBkhlJ9},
}
Figure