TL;DR
This paper introduces a novel "Conformalized Abstention Policy" (CAP) that allows Large Language Models (LLMs) and Vision-Language Models (VLMs) to intelligently abstain from answering when they are uncertain. By combining reinforcement learning with conformal prediction, CAP dynamically adjusts abstention thresholds, leading to improved accuracy, better hallucination detection, and more reliable uncertainty quantification compared to existing methods.
What Problem Does This Paper Address?
LLMs and VLMs are increasingly deployed in safety-critical applications, but their "black box" nature makes it hard to assess their reliability and manage risks. While uncertainty quantification (UQ) methods like conformal prediction (CP) can help identify uncertain predictions, traditional CP relies on static thresholds. This inflexibility means they can't adapt to varying task complexities or data distributions, leading to suboptimal trade-offs: either overly narrow prediction sets that miss correct answers (sacrificing coverage) or excessively large sets that are uninformative. This paper aims to bridge this gap by developing an adaptive abstention policy that balances accuracy, coverage, and informativeness.
Methods
The core innovation is Conformalized Abstention Policy (CAP), which frames abstention as a reinforcement learning (RL) problem.
- Conformal Prediction (CP): The foundation is CP, which provides statistical guarantees on prediction coverage. It generates prediction sets that are guaranteed to contain the true label with a certain probability (e.g., 90%).
- Adaptive Thresholds: Instead of a single threshold for CP, CAP introduces two adaptive thresholds:
Îpredictandĝabstain. These thresholds dynamically partition nonconformity scores into three regimes: - Single Best Prediction: Low uncertainty, confident prediction.
- Set Prediction: Moderate uncertainty, multiple plausible answers.
- Abstain: High uncertainty, deferring the decision.
- Reinforcement Learning (RL): A policy network (using the REINFORCE algorithm) is trained to dynamically adjust these
αandβthresholds (which defineÎpredictandĝabstain). The policy learns to optimize a cost function that balances accuracy, set size, coverage, and abstention. This allows the model to learn optimal abstention strategies based on the observed data and task.
Datasets
The authors evaluated their approach on a diverse set of ten benchmark datasets for both VLMs and LLMs.
For VLMs: * MMBench * OODCV-VQA * ScienceQA * SEEDBench * AI2D
For LLMs: * MMLU * CosmosQA * HellaSwag * HaluDial * HaluSum
All datasets were standardized to have multiple-choice options (typically four or six, with "I don't know" and "None of the above" added for consistency).
Key Insights and Ideas
- Dynamic Abstention is Crucial: Static UQ methods are insufficient for complex, real-world LLM/VLM applications. Adaptive abstention, learned through RL, is key to managing risk effectively.
- Balancing Act: CAP successfully balances competing objectives: high accuracy, guaranteed coverage, informative prediction sets, and intelligent abstention. This is a significant improvement over baselines that often sacrifice one for another.
- Beyond Binary Abstention: The three-regime decision (single, set, abstain) offers a more nuanced approach than a simple predict/abstain choice.
- RL for UQ Tuning: Framing the tuning of CP thresholds as an RL problem is a powerful way to optimize for complex, multi-objective performance metrics.
Key Results
- Hallucination Detection: CAP improved AUROC (hallucination detection) by up to 22.19% compared to baselines.
- Uncertainty-Guided Generation: CAP boosted AUARC (uncertainty-guided selective generation) by up to 21.17%.
- Calibration: CAP achieved significantly lower Expected Calibration Error (ECE) – an average reduction of 82.9% across VLMs and LLMs compared to APS, and 74.8% compared to LAC.
- Coverage: CAP consistently met the 90% coverage guarantee across all datasets and models.
- Accuracy: CAP achieved higher average accuracy than APS and LAC across various models and datasets.
- Set Size: CAP struck a balance, producing smaller, more controlled prediction sets than APS, while avoiding the overly narrow sets of LAC that lead to under-coverage.
Limitations and Future Work
- Overfitting and Bias: The authors acknowledge that learned policies might overfit to specific datasets or introduce biases, potentially distorting CP's theoretical guarantees.
- Parameter Tuning: CAP introduces additional parameters and relies on well-tuned reward functions, which might require careful optimization for different data distributions.
- Distribution Shifts: Extreme distribution shifts or limited calibration data can impact performance. The authors suggest future work could involve enforcing distribution-aware regularization and calibrating policies through out-of-sample validation.
Summary updated