Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement

Gwenyth Portillo Wightman, Alexandra Delucia, Mark DredzeACL 2023

TL;DR

Large language models (LLMs) are great at many tasks but struggle to provide reliable confidence scores for their predictions. This paper proposes a novel method: instead of relying on a single prompt, they use prompt agreement across multiple diverse prompts to estimate confidence. This approach leads to more calibrated and trustworthy confidence scores, crucial for integrating LLMs into real-world applications.

What Problem Does This Paper Address?

Modern LLMs can perform a wide range of tasks, including classification, by framing them as text generation problems. This allows for "few-shot learning" where models can adapt to new tasks with just a few examples and a well-crafted prompt. However, a significant challenge arises when users need to know how confident the LLM is in its prediction. Traditional classifiers often output probabilities for each label, which can be well-calibrated. LLMs, on the other hand, generate text, and their confidence is often tied to the log probability of that generation. This log probability doesn't always align with the actual probability of the prediction being correct, leading to poorly calibrated confidence scores. This lack of reliable confidence makes it difficult to trust LLM outputs in high-stakes scenarios like healthcare or finance, or to integrate them effectively into larger decision-making systems.

Methods

The core idea of this paper is "prompt agreement." Instead of relying on a single prompt to elicit an answer and its associated confidence score, the authors propose using multiple, diverse prompts for the same input instance. They then measure the consistency of the model's responses across these prompts to estimate confidence.

Two primary methods for measuring this agreement are proposed:

  1. Log Probabilities Across Prompts: For each instance, the model is prompted multiple times, and the log probability of the predicted answer is recorded for each. The majority label across these prompts is identified, and its confidence score is calculated as the average of the log probabilities for that majority label.
  2. Answer Agreement Across Multiple Prompts (Rand Index): This method focuses on the diversity of answers. It counts how many times each unique answer appears across the different prompts. The Rand index is then used to quantify the agreement between the observed "clustering" of prompts (based on their answers) and an "ideal" clustering where all prompts yield the same answer. A higher Rand index indicates greater agreement and thus higher confidence.

These methods are compared against a baseline of using the log probability from a single, official task prompt.

Datasets

The authors evaluate their methods on ten multiple-choice question-answering (QA) datasets. These datasets are drawn from the T0 training mixture and include:

  • CoS-E v1.11
  • Cosmos QA
  • DREAM
  • QASC
  • Quail
  • Quarel
  • Quartz
  • SciQ
  • Social IQA
  • WIQA

They use the validation splits of these datasets.

Key Insights and Ideas

  • Consistency is Key: The fundamental insight is that a model's confidence in an answer is related to how consistently it produces that answer across different ways of asking the question. If a model consistently picks "eggplant" when asked about a fruit, even with varied phrasing, it's likely more confident than if it sometimes says "eggplant" and sometimes "potato."
  • Beyond Log Probability: The paper demonstrates that simply relying on the log probability of a single generation is insufficient for accurate confidence estimation. LLMs can assign high probability to semantically similar but distinct answers (e.g., "Answer A" vs. "The answer is A"), which can be misleading.
  • Human-Authored Prompts are Powerful: The study highlights that diverse, human-authored prompts lead to the most calibrated confidence scores, suggesting that human intuition in crafting prompts is valuable.
  • Automated Prompts Show Promise (with Caveats): While automatically generated prompts (AGPs) can improve confidence over single prompts, their performance is mixed. The quality of the AGPs seems to be a critical factor, and further research is needed to optimize their generation for better confidence estimation.
  • Accuracy and Calibration are Linked: The paper observes a correlation between higher task accuracy and greater improvements in calibration when using multiple prompts.

Key Results

  • Improved Calibration: Using multiple prompts (both human-authored and automatically generated) consistently leads to better calibrated confidence scores compared to using a single prompt, as measured by Expected Calibration Error (ECE) and Swapped Pairs metrics.
  • Human Prompts Outperform: The "Multiple / agreement" method using human-authored prompts generally yields the most calibrated confidence scores across models and datasets.
  • Model Differences: T0++ and GPT-3 show better inherent calibration than FLAN-T5-XXL, but the prompt agreement method significantly reduces the calibration gap for all models.
  • Accuracy Gains: Using multiple prompts consistently leads to higher task accuracies across all models compared to single prompts.
  • AGP Performance: While AGPs show improvement over single prompts, they do not consistently outperform human-authored prompts. However, they still offer a viable alternative when human prompts are unavailable.

Limitations and Future Work

  • Lack of Human Evaluation: The authors acknowledge that a significant limitation is the absence of human evaluation. Since confidence scores are often used for explainability, a practical evaluation would involve users making decisions based on these scores.
  • Focus on Multiple-Choice QA: The study is limited to multiple-choice QA tasks. While this is representative of classification, it doesn't cover open-ended generation tasks like summarization.
  • Model Size: The study focused on the largest models in each family. Replicating the study with smaller models would be valuable to understand the impact of parameter count on the method's robustness.
  • Rand Index Normalization: The Rand index scores are not directly interpretable as probabilities, and while normalization methods exist, they were avoided in this evaluation to maintain a fair comparison.
  • Automated Prompt Generation: The authors suggest that future work could explore more advanced prompt generation methods to improve the quality and effectiveness of AGPs for confidence estimation.

Summary updated

BibTeX
@inproceedings{wightman2023strength,
  author = {Gwenyth Portillo Wightman and Alexandra Delucia and Mark Dredze},
  title = {{Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure