TL;DR
This paper introduces SIMBA UQ, a novel framework for assessing how confident Large Language Models (LLMs) are in their generated outputs. SIMBA UQ leverages the similarity between multiple generated responses to estimate confidence, offering a "black-box" approach that doesn't require internal model access and can be applied to diverse tasks. The authors demonstrate that their similarity-based aggregation methods yield better calibrated confidences compared to existing baselines across question answering, summarization, and text-to-SQL tasks.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly powerful, but a critical challenge is knowing when they "don't know" what they're talking about. This is where Uncertainty Quantification (UQ) comes in, aiming to provide measures of confidence in an LLM's output. Traditional UQ methods often require access to the LLM's internal workings (white-box), which is not always feasible or desirable due to model proprietary nature or rapid evolution. Black-box UQ methods, which only use the model's outputs, are thus highly valuable. A common assumption in black-box UQ is the "consistency hypothesis": more consistent generations are more likely to be correct. However, effectively leveraging this consistency across complex generative tasks, especially when LLMs are asked to express confidence directly (verbalized UQ), can be problematic, often leading to overconfidence. This paper addresses the need for robust, black-box UQ methods that can reliably estimate confidence by focusing on the consistency of generated outputs without relying on the LLM to explicitly state its confidence.
Methods
The core of this paper is the SIMBA UQ (Similarity-Based Aggregation) framework. It operates in three main phases:
- Sampling/Generation: Multiple outputs (samples) are generated by the LLM for a given input query, often by varying sampling temperatures.
- Pairwise Similarity Computation: A chosen similarity metric (e.g., Jaccard, Rouge-L) is used to calculate the similarity between every pair of generated samples.
- Similarity Aggregation: This is the key novel contribution. Instead of clustering outputs or directly asking the LLM for confidence, SIMBA UQ aggregates the pairwise similarities to derive a confidence score for each generation. The paper proposes three main aggregation strategies:
- Simple Aggregation: Uses an arithmetic mean of pairwise similarities (or distances) as a proxy for confidence. This is mathematically equivalent to some spectral clustering approaches.
- Bayesian Aggregation: Treats pairwise similarities as evidence to update a prior belief about a generation's correctness. This approach requires a small labeled dataset for training.
- Aggregation by Classification: Frames confidence estimation as a classification task. Pairwise similarities (and potentially other features like generative scores) are used as input features to train a probabilistic classifier (e.g., random forest) to predict the probability of a generation being correct. This also requires a small training set.
The paper emphasizes that these methods are primarily black-box, though they can incorporate white-box features if available.
Datasets
The authors evaluated their methods on a diverse set of datasets across three key NLP tasks:
- Question Answering (QA): CoQA, TriviaQA, and Natural Questions.
- Summarization: XSum, SamSum, and CNN Dailymail.
- Text-to-SQL: Spider, Spider-Realistic, and BIRD.
They sub-selected the first 1000 queries for QA and summarization tasks for their experiments. The BIRD dataset is highlighted as a recent cross-domain benchmark for text-to-SQL.
Key Insights and Ideas
- Non-verbalized confidence is key: The paper argues that aggregating pairwise similarities directly, rather than relying on LLMs to verbalize their confidence, avoids issues like overconfidence and is more robust for complex tasks.
- Consistency hypothesis holds practical value: The core assumption that consistent generations are more likely to be correct is empirically validated and forms the basis for their similarity aggregation.
- Classification as a powerful aggregation tool: Framing confidence estimation as a classification problem, using similarities as features, proved to be a highly effective and robust approach, often outperforming simpler aggregation methods.
- Generalizability across tasks: The SIMBA UQ framework and its proposed methods demonstrate effectiveness across diverse tasks like QA, summarization, and text-to-SQL, including structured outputs like SQL queries.
- Limited data is sufficient: The classification and Bayesian aggregation methods require only a small amount of labeled data for training, making them practical for many real-world scenarios.
Key Results
The paper presents extensive experimental results across multiple datasets and models. Key findings include:
- Improved Calibration: The proposed SIMBA UQ methods, particularly the classification-based approaches (
clf-pairs,clf-mean+gen,clf-pairs+gen), consistently achieved lower Adaptive Calibrated Error (ACE), indicating better calibrated confidence estimates compared to baselines likearith-aggandspec-ecc. - Competitive Performance: While not always outperforming baselines on Accuracy from Top Selection (ATS) and Area Under the ROC Curve (AUROC), the proposed methods showed strong performance across these metrics as well, especially when using random forests for classification.
- Robustness to Similarity Metrics: Experiments on the BIRD dataset showed that common text/token similarity metrics like Jaccard and Rouge-L are suitable for UQ even for SQL generation, and the classification approach remained high-performing regardless of the chosen metric.
- Effectiveness of Classification: Ablation studies showed that classification with random forests was more robust to the number of samples compared to unsupervised methods like
arith-agg.
Limitations and Future Work
- Assumption of No Data Uncertainty: The methods assume no inherent uncertainty in the ground truth data, which might not always hold.
- Conditional Independence: The Bayesian aggregation method relies on a conditional independence assumption that may not always be true, potentially impacting its performance.
- Moderate Gains on ATS/AUROC: While calibration improved significantly, gains on ATS and AUROC were sometimes moderate, suggesting room for further improvement in distinguishing correct from incorrect responses.
- Out-of-Domain Performance: The current experiments focus on in-domain settings. Investigating out-of-domain performance and domain adaptation for these methods is a crucial area for future work.
- Theoretical Justification: Further theoretical work is needed to fully understand when the core consistency assumption holds and to provide stronger guarantees for individual instance confidence estimates.
Summary updated