TL;DR
CARE (Cross-question Abstention Reward Estimation) is a novel reinforcement learning framework that trains Large Language Models (LLMs) to verbalize reliable confidence scores by comparing answers across a batch rather than in isolation. By optimizing for "coverage at risk," CARE enables models to effectively abstain from difficult questions, boosting reliable task coverage on math benchmarks from a negligible 0.8% to over 42%.
What Problem Does This Paper Address?
As LLMs are deployed in high-stakes domains like autonomous driving or medical assistance, their ability to abstain—to say "I don't know"—becomes as important as their ability to answer correctly. Ideally, a model should provide a confidence score that allows a user to set a "risk tolerance" (e.g., "only give me answers where you are 95% sure").
Current Reinforcement Learning (RL) methods for confidence estimation typically use pointwise rewards. They reward the model for outputting a confidence of 1 if the answer is correct and 0 if it is incorrect. While this sounds logical, it leads to "confidence collapse": the model learns to output only 0s and 1s. This polarization destroys the ranking information needed to distinguish between "mostly sure" and "completely sure" answers, making it impossible to tune the model for specific risk levels.
Methods
The authors introduce CARE, which shifts the learning signal from individual question-answer pairs to a set of pairs within a training batch.
- Cross-Question Thresholding: Instead of a fixed target of 0 or 1, CARE calculates a dynamic abstention threshold based on the model's current performance across a batch. For a given risk tolerance (e.g., 5% error rate), it finds the confidence level that would have allowed the model to answer the most questions while staying under that error limit.
- Relative Reinforcement: Each answer is rewarded based on which side of this dynamic threshold its confidence falls. A correct answer is rewarded for being above the threshold, and an incorrect one for being below it. This forces the model to rank its own answers relative to each other.
- Diversity Rewards: To prevent the model from defaulting to a few "safe" numbers (like 0.8 or 0.9), CARE includes entropy and uniqueness rewards. These encourage the model to use a fine-grained, continuous range of confidence scores.
Datasets
The researchers evaluated CARE across two primary training regimes:
- Math: Training on Big-Math-Digits and evaluating on MATH-500 and GSM8K.
- General Knowledge/Reasoning: Training on HotpotQA (multi-hop Wikipedia QA) and testing for out-of-distribution (OOD) generalization on CommonsenseQA, GPQA, SimpleQA, and TriviaQA.
Key Insights and Ideas
- Ranking over Absolute Values: The "aha moment" is that for selective prediction, the absolute value of a confidence score matters less than its rank relative to other answers. By training on the ranking, CARE preserves the resolution needed for low-risk applications.
- Multi-Risk Training: By training on multiple risk tolerances simultaneously (e.g., 1%, 5%, 15%), the model learns a more robust confidence distribution that generalizes across different user needs.
- On-Policy Advantage: The authors found that using a "rolling buffer" of old answers (off-policy) actually hurt performance. Estimating thresholds works best when using the model's most recent "on-policy" generations, as they accurately reflect its current capabilities.
- Verbalization vs. Logits: Unlike methods that rely on internal model probabilities (logits), CARE works with verbalized confidence (the model literally writes "Confidence: 0.85"). This makes it applicable even when internal probabilities are unavailable or distorted by techniques like Chain-of-Thought reasoning.
Key Results
The quantitative improvements are striking, particularly at low risk levels where previous methods failed completely:
- Math Benchmarks: On a suite of math tasks, CARE increased the Coverage at 5% risk (C@5) from 0.8% to 42.4% compared to standard RL methods.
- OOD Generalization: When trained on HotpotQA and tested on unseen general knowledge tasks, CARE improved C@5 from 12.3% to 23.6%.
- Smoother Risk-Coverage Curves: As shown in Figure 3, CARE produces much smoother curves, meaning users can reliably pick any risk tolerance and get predictable behavior from the model.
Limitations and Future Work
- OOD Challenges: While CARE improves OOD generalization, there is still a significant gap between in-distribution and out-of-distribution performance. The model can still be "surprised" by entirely new domains.
- Accuracy vs. Confidence: The authors note that while CARE doesn't hurt accuracy, it doesn't primarily aim to increase it. Future work could explore combining CARE with accuracy-maximizing objectives to get the best of both worlds.
- Computational Overhead: Calculating cross-question rewards requires larger batch sizes during training to get stable threshold estimates, which may increase memory requirements.
Summary updated