Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?

Shiyu Ni, Keping Bi, Lulu Yu, Jiafeng Guo

Abstract

Large Language Models (LLMs) often struggle to communicate their internal certainty, leading to hallucinations. This research investigates whether internal token probabilities or verbalized self-reports are more "honest" indicators of a model's knowledge. By evaluating models like Llama2, Mistral, and GPT-3.5, we find that probabilistic confidence is a significantly more reliable predictor of accuracy than verbalized confidence. However, a "translation gap" exists where models fail to accurately express their internal "gut feelings" in natural language, often exhibiting verbalized overconfidence.

Introduction

The reliability of Large Language Models (LLMs) hinges on their ability to recognize their own knowledge boundaries. When pushed beyond these limits, models frequently hallucinate. To address this, researchers evaluate self-awareness through two primary lenses: Probabilistic Confidence (internal log-probabilities) and Verbalized Confidence (direct natural language assertions of certainty). This paper explores the relationship between these two methods and determines which provides a more truthful representation of model knowledge.

Methods

The study evaluates four models: Llama2-7B-Chat, Mistral-7B-Instruct, GPT-3.5-Turbo (ChatGPT), and GPT-3.5-Turbo-Instruct.

  • Probabilistic Confidence: Calculated using average log-probability (perplexity) of generated answers, converted to binary labels via an optimal threshold determined on a validation set.
  • Verbalized Confidence: Models were prompted to append "certain" or "uncertain" to their responses.
  • Analysis: Correlation (Spearman/Kendall) and binning techniques were used to measure the alignment between internal signals and verbalized output.

Datasets

Two primary benchmarks were utilized:

  1. Natural Questions (NQ): Real-world factual queries from Google search data.
  2. Parent-Child Dataset: Designed to test the impact of question frequency by pairing common facts (e.g., "Who is the parent of X?") with less frequent inverse facts (e.g., "Who is the child of Y?").

Key Insights

The Translation Gap and Overconfidence

There is a significant disconnect between internal probability and verbalized output. Models are consistently "cocky" when expressing certainty in words, overestimating their correctness compared to their internal signals. This suggests that LLMs possess a "gut feeling" they cannot yet accurately articulate.

The Frequency Paradox

LLMs demonstrate a better perception of their knowledge boundaries on less frequent (unfamiliar) questions. While they tend to be overconfident on common questions, their internal probabilities drop sharply on obscure queries, indicating a clearer grasp of their own ignorance.

Table 2: Performance comparison between Parent (frequent) and Child (infrequent) datasets for GPT models.

Calibration Requirements

Probabilistic confidence, while more accurate, requires dataset-specific calibration. The optimal threshold for "certainty" varies significantly across different domains, making it less flexible than verbalized confidence in zero-shot scenarios.

Key Results

Probabilistic perception consistently outperformed verbalized confidence across all tested models. As shown in the table below, alignment scores—which measure how well confidence matches actual correctness—are markedly higher for probabilistic strategies than for verbalized ones.

Table 1: Comparison of accuracy and alignment across different models and confidence strategies.

For instance, on the NQ dataset, Llama2 achieved an alignment of ~72.5% using probabilities, compared to only ~45.1% when verbalizing confidence. Similarly, ChatGPT's probabilistic alignment jumped to 88.2% on the "Child" dataset, highlighting the model's increased sensitivity to its own uncertainty in unfamiliar contexts.

Limitations and Future Work

A major hurdle for implementing probabilistic confidence is the "black-box" nature of many commercial APIs, which do not provide token-level log-probabilities. Additionally, the need for a validation set to tune thresholds limits flexibility. Future research should focus on calibration training to help models better interpret their internal probability distributions and express that certainty accurately in natural language, bridging the gap between what they "feel" and what they say.

Summary updated

Abstract

Large language models (LLMs) have been found to produce hallucinations when the question exceeds their internal knowledge boundaries. A reliable model should have a clear perception of its knowledge boundaries, providing correct answers within its scope and refusing to answer when it lacks knowledge. Existing research on LLMs’ perception of their knowledge boundaries typically uses either the probability of the generated tokens or the verbalized confidence as the model’s confidence in its response. However, these studies overlook the differences and connections between the two. In this paper, we conduct a comprehensive analysis and comparison of LLMs’ probabilistic perception and verbalized perception of their factual knowledge boundaries. First, we investigate the pros and cons of these two perceptions. Then, we study how they change under questions of varying frequencies. Finally, we measure the correlation between LLMs’ probabilistic confidence and verbalized confidence. Experimental results show that 1. LLMs’ probabilistic perception is generally more accurate than verbalized perception but requires an in-domain validation set to adjust the confidence threshold. 2. Both perceptions perform better on less frequent questions. 3. It is challenging for LLMs to accurately express their internal confidence in natural language.
BibTeX
@inproceedings{ni2024are,
  author = {Shiyu Ni and Keping Bi and Lulu Yu and Jiafeng Guo},
  title = {{Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?}},
  booktitle = {CCIR},
  year = {2024},
  url = {https://openreview.net/forum?id=HYWTdJEjoZ},
}
Figure