Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models

Tobias Groot, Matias Valdenegro-ToroNAACL 2024

TL;DR

This paper investigates how well Large Language Models (LLMs) and Vision-Language Models (VLMs) can express their own uncertainty. The study reveals that most models are overconfident and struggle to accurately gauge their reliability, even with specific prompting. This highlights a critical gap in current AI capabilities, suggesting that simply increasing model size or performance doesn't automatically lead to better self-awareness.

What Problem Does This Paper Address?

As AI models like LLMs and VLMs become more sophisticated and integrated into various applications, their reliability is paramount. A key aspect of reliability is understanding when a model is likely to be wrong – its uncertainty. Ideally, a model's expressed confidence should correlate with its accuracy: high confidence for correct answers, low confidence for incorrect ones. However, current LLMs and VLMs often "hallucinate" or produce incorrect information, and it's unclear if they can reliably signal when they are unsure. This paper addresses the critical need to evaluate and understand the verbalized uncertainty estimation capabilities of these advanced models, particularly in the context of their increasing deployment.

Methods

The researchers evaluated four LLMs (GPT-4, GPT-3.5, LLaMA-2-70b, PaLM 2) and two VLMs (GPT-4V, Gemini Pro Vision). They employed a prompting strategy to elicit verbalized uncertainty from these models across various tasks. For LLMs, this included sentiment analysis, math word problems, and named-entity recognition. For VLMs, an image recognition task was designed. A novel metric, the Net Calibration Error (NCE), was introduced to assess not only the degree of miscalibration but also its direction (overconfidence vs. underconfidence), complementing the standard Expected Calibration Error (ECE).

Datasets

  • Sentiment Analysis: Stanford Sentiment Treebank (SST) dataset (both SST2 with binary labels and the original SST with float labels).
  • Math Word Problems: GSM8K dataset.
  • Named-Entity Recognition: CoNLL 2003 dataset (English and German sentences).
  • Image Recognition: A newly created dataset called the Japanese Uncertain Scenes (JUS) dataset. This dataset comprises 39 images with challenging prompts, including object counting and difficult queries, specifically designed to test VLM uncertainty estimation.

Key Insights and Ideas

  • Overconfidence is the Norm: Across most tasks and models, LLMs and VLMs tend to be overconfident, meaning they express higher confidence than their accuracy warrants. This is a significant finding, as it suggests current models lack robust self-awareness.
  • Task-Dependent Uncertainty: The degree and direction of miscalibration vary by task. For instance, sentiment analysis showed a tendency towards underconfidence, while math word problems and named-entity recognition exhibited strong overconfidence.
  • VLMs Struggle with Image Uncertainty: Similar to LLMs, VLMs also show limited accuracy in estimating their uncertainty for image recognition tasks, with a predominant trend towards overconfidence.
  • GPT-4 and GPT-4V Show Relative Improvement: While still exhibiting overconfidence, GPT-4 and GPT-4V generally demonstrated better calibration compared to their counterparts, suggesting architectural or training differences might play a role.
  • Novel Metric for Direction: The introduction of the Net Calibration Error (NCE) is a valuable contribution, allowing researchers to distinguish between overconfidence and underconfidence, which is crucial for understanding model behavior.

Key Results

  • LLMs: In sentiment analysis, GPT-3.5 showed the best calibration with a small ECE of 3.55%. For math word problems, only GPT-4 was well-calibrated; other models were highly overconfident with ECEs as high as 74.8% (GPT-3.5). In named-entity recognition, GPT-4 was less overconfident than GPT-3.5.
  • VLMs: For image recognition, GPT-4V had a significantly lower ECE (11.3) compared to Gemini Pro Vision (38.4), indicating better calibration, though both were overconfident.
  • JUS Dataset: The JUS dataset proved effective in highlighting VLM limitations, with models often producing incorrect verbalized uncertainty, even for simple counting tasks.

Limitations and Future Work

  • Dataset Size: The JUS dataset, while effective, is small (39 images), limiting the generalizability of VLM findings.
  • Task Iterations: Each task was performed only once per model, which might not capture performance variability. Future work could involve multiple iterations.
  • Model Selection: The study focused on a select group of LLMs and VLMs, not the entire landscape of available models.
  • Future Directions: The authors suggest exploring how techniques like "Chain of Thought" (CoT) prompting might improve uncertainty estimation. They also highlight the potential for future research on open-source models like LLaMA-2-70b to investigate direct model modifications for better uncertainty calibration. The development of novel VLM uncertainty evaluation methods is also encouraged.

Summary updated

BibTeX
@inproceedings{groot2024overconfidence,
  author = {Tobias Groot and Matias Valdenegro-Toro},
  title = {{Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure