Cross-Question Reliable Reinforcement Learning

Hector G. Rodriguez, Marcus RohrbacharXiv 2026 · arXiv

TL;DR

CARE (Cross-question Abstention Reward Estimation) is a novel reinforcement learning framework that trains Large Language Models (LLMs) to verbalize reliable confidence scores by comparing answers across a batch rather than in isolation. By optimizing for "coverage at risk," CARE enables models to effectively abstain from difficult questions, boosting reliable task coverage on math benchmarks from a negligible 0.8% to over 42%.

What Problem Does This Paper Address?

As LLMs are deployed in high-stakes domains like autonomous driving or medical assistance, their ability to abstain—to say "I don't know"—becomes as important as their ability to answer correctly. Ideally, a model should provide a confidence score that allows a user to set a "risk tolerance" (e.g., "only give me answers where you are 95% sure").

Current Reinforcement Learning (RL) methods for confidence estimation typically use pointwise rewards. They reward the model for outputting a confidence of 1 if the answer is correct and 0 if it is incorrect. While this sounds logical, it leads to "confidence collapse": the model learns to output only 0s and 1s. This polarization destroys the ranking information needed to distinguish between "mostly sure" and "completely sure" answers, making it impossible to tune the model for specific risk levels.

Methods

The authors introduce CARE, which shifts the learning signal from individual question-answer pairs to a set of pairs within a training batch.

  1. Cross-Question Thresholding: Instead of a fixed target of 0 or 1, CARE calculates a dynamic abstention threshold based on the model's current performance across a batch. For a given risk tolerance (e.g., 5% error rate), it finds the confidence level that would have allowed the model to answer the most questions while staying under that error limit.
  2. Relative Reinforcement: Each answer is rewarded based on which side of this dynamic threshold its confidence falls. A correct answer is rewarded for being above the threshold, and an incorrect one for being below it. This forces the model to rank its own answers relative to each other.
  3. Diversity Rewards: To prevent the model from defaulting to a few "safe" numbers (like 0.8 or 0.9), CARE includes entropy and uniqueness rewards. These encourage the model to use a fine-grained, continuous range of confidence scores.
Figure 1: Cross-Question Abstention Reward Estimation (CARE). LLMs are trained to verbalize confidence in their answers. Existing pointwise rewards score predicted confidence only against its correctness label, i.e. confidence reward Ri(ai, ci) for answer ai depends only on that answer’s correctness and confidence prediction. Left: CARE uses sets of answers and confidences S across a batch to reinforce coverage at risk, so the reward Ri(ai, ci, cj∈S) is informed by other confidence predictions cj∈S. Right: C@5 on unseen math benchmarks and on HotpotQA-to-OOD transfer.

Datasets

The researchers evaluated CARE across two primary training regimes:

  • Math: Training on Big-Math-Digits and evaluating on MATH-500 and GSM8K.
  • General Knowledge/Reasoning: Training on HotpotQA (multi-hop Wikipedia QA) and testing for out-of-distribution (OOD) generalization on CommonsenseQA, GPQA, SimpleQA, and TriviaQA.

Key Insights and Ideas

  • Ranking over Absolute Values: The "aha moment" is that for selective prediction, the absolute value of a confidence score matters less than its rank relative to other answers. By training on the ranking, CARE preserves the resolution needed for low-risk applications.
  • Multi-Risk Training: By training on multiple risk tolerances simultaneously (e.g., 1%, 5%, 15%), the model learns a more robust confidence distribution that generalizes across different user needs.
  • On-Policy Advantage: The authors found that using a "rolling buffer" of old answers (off-policy) actually hurt performance. Estimating thresholds works best when using the model's most recent "on-policy" generations, as they accurately reflect its current capabilities.
  • Verbalization vs. Logits: Unlike methods that rely on internal model probabilities (logits), CARE works with verbalized confidence (the model literally writes "Confidence: 0.85"). This makes it applicable even when internal probabilities are unavailable or distorted by techniques like Chain-of-Thought reasoning.

Key Results

The quantitative improvements are striking, particularly at low risk levels where previous methods failed completely:

  • Math Benchmarks: On a suite of math tasks, CARE increased the Coverage at 5% risk (C@5) from 0.8% to 42.4% compared to standard RL methods.
  • OOD Generalization: When trained on HotpotQA and tested on unseen general knowledge tasks, CARE improved C@5 from 12.3% to 23.6%.
  • Smoother Risk-Coverage Curves: As shown in Figure 3, CARE produces much smoother curves, meaning users can reliably pick any risk tolerance and get predictable behavior from the model.
Risk-coverage curves showing CARE (green) consistently outperforming baselines across various risk levels.

Limitations and Future Work

  • OOD Challenges: While CARE improves OOD generalization, there is still a significant gap between in-distribution and out-of-distribution performance. The model can still be "surprised" by entirely new domains.
  • Accuracy vs. Confidence: The authors note that while CARE doesn't hurt accuracy, it doesn't primarily aim to increase it. Future work could explore combining CARE with accuracy-maximizing objectives to get the best of both worlds.
  • Computational Overhead: Calculating cross-question rewards requires larger batch sizes during training to get stable threshold estimates, which may increase memory requirements.

Summary updated

Abstract

AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.
BibTeX
@misc{rodriguez2026cross,
  author = {Hector G. Rodriguez and Marcus Rohrbach},
  title = {{Cross-Question Reliable Reinforcement Learning}},
  year = {2026},
  eprint = {2611.25292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25292},
}
Figure