Abstract
For large language models (LLMs) to be trusted by humans they need to be well-calibrated in the sense that they can accurately assess and communicate how likely it is that their predictions are correct. Recent work has focused on the quality of internal LLM confidence assessments, but the question remains of how well LLMs can communicate this internal model confidence to human users. This paper explores the disparity between external human confidence in an LLM's responses and the internal confidence of the model. Through experiments involving multiple-choice questions, we systematically examine human users' ability to discern the reliability of LLM outputs. Our study focuses on two key areas: (1) assessing users' perception of true LLM confidence and (2) investigating the impact of tailored explanations on this perception. The research highlights that default explanations from LLMs often lead to user overestimation of both the model's confidence and its' accuracy. By modifying the explanations to more accurately reflect the LLM's internal confidence, we observe a significant shift in user perception, aligning it more closely with the model's actual confidence levels. This adjustment in explanatory approach demonstrates potential for enhancing user trust and accuracy in assessing LLM outputs. The findings underscore the importance of transparent communication of confidence levels in LLMs, particularly in high-stakes applications where understanding the reliability of AI-generated information is essential.
Introduction
As Large Language Models (LLMs) become integral to decision-making, the "trust but verify" mantra is increasingly difficult to uphold. For effective human-AI collaboration, a model must be well-calibrated: its expressed confidence should match its actual probability of being correct. However, a significant "calibration gap" exists where users consistently overestimate LLM accuracy. This is often driven by assertive, verbose explanations that mask the model's internal uncertainty.
This research addresses two primary challenges:
- The Calibration Gap: The discrepancy between a human's confidence in an LLM's answer and the model's actual accuracy.
- The Discrimination Gap: The inability of humans to distinguish between a model's correct and incorrect answers based solely on its written explanation.
Methodology
The researchers conducted behavioral experiments with over 300 human participants to evaluate how different explanation styles influence trust. The study utilized two primary benchmarks: MMLU (Massive Multitask Language Understanding) for multiple-choice questions and Trivia QA for short-answer tasks.
Internal model confidence was measured using token likelihoods for multiple-choice questions and the "pTrue" method—prompting the model to evaluate its own answer—for short-answer questions. The core of the experiment involved manipulating explanations across two dimensions: uncertainty language (ranging from "I am not sure" to "I am sure") and length (Long, Short, or Uncertainty-only).
Key Insights
The study revealed several critical biases in how humans process AI-generated information:
- The "Length Bias": Participants reported significantly higher confidence in longer explanations, regardless of their factual accuracy. This "verbosity bias" suggests humans use length as a heuristic for quality.
- The Assertiveness Trap: LLMs, often optimized via Reinforcement Learning from Human Feedback (RLHF) to be helpful and persuasive, tend to generate overly convincing rationales even for hallucinations or errors.
- Shallow Processing: Instead of deeply evaluating the logic of an explanation, humans often rely on superficial "textual cues" like tone and length.
- Verbal Hedging as a Solution: The researchers found that simply prepending a verbal hedge that matches the model's internal probability score (e.g., "I'm not entirely sure, but...") effectively aligns human perception with model reality.
Results and Impact
The results demonstrate that default LLM outputs fail to calibrate human users. For GPT-3.5, the human Expected Calibration Error (ECE) was 0.264 (indicating high overconfidence), and the ability to discriminate between right and wrong answers (AUC) was only 0.589—barely better than chance.
However, when explanations were tailored to reflect internal confidence, the calibration gap narrowed significantly. For the same model, human ECE dropped to 0.150, and the discrimination AUC improved to 0.694. Notably, self-rated expertise in a topic did not improve a participant's ability to identify LLM errors, highlighting the universal nature of these cognitive biases.
Limitations and Future Directions
While effective for "factoid" questions, it remains to be seen if these findings extend to complex reasoning, creative writing, or coding. Furthermore, the current two-step process—calculating confidence then generating a tailored explanation—could be streamlined into a single-step generation process. Finally, the authors call for new alignment strategies that prioritize honest uncertainty communication over mere persuasiveness, addressing the inherent tension created by current RLHF practices.
Summary updated