Why Language Models Hallucinate

Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin ZhangarXiv 2025 · DOI

TL;DR

Language models often "hallucinate" by generating plausible but incorrect information instead of admitting uncertainty. This paper argues that this behavior stems from the training and evaluation processes that reward confident guessing over honest uncertainty. They propose a socio-technical solution: modifying existing benchmarks to incentivize accurate uncertainty expression, rather than solely focusing on performance metrics.

What Problem Does This Paper Address?

Large language models (LLMs) are prone to "hallucinations," producing confident, false statements. This undermines their trustworthiness and utility. While the phenomenon is well-known, its root causes and persistence, even in state-of-the-art models, remain a challenge. The authors argue that current training and evaluation paradigms inadvertently encourage LLMs to guess when uncertain, leading to this problematic behavior. They aim to demystify hallucinations by analyzing their statistical origins and proposing a path towards more trustworthy AI.

Methods

The paper frames hallucination as a problem of statistical learning theory, specifically connecting it to binary classification. They introduce an "Is-It-Valid" (IIV) classification problem to analyze generative errors. A key contribution is a theoretical reduction showing that the generative error rate (hallucinations) is directly related to the IIV misclassification rate. This framework allows them to analyze errors arising from both pretraining (learning the data distribution) and post-training (fine-tuning). They also introduce a formalization of "arbitrary facts" and "singleton rate" to model specific types of hallucinations.

Datasets

The paper does not introduce new datasets. Instead, it analyzes the behavior of existing language models on various prompts and discusses how common benchmarks are used for evaluation. Specific models mentioned include ChatGPT, DeepSeek-V3, Meta AI, and Claude 3.7 Sonnet.

Key Insights and Ideas

  • Hallucinations are not mysterious: They arise naturally from statistical pressures in training and evaluation, akin to errors in binary classification.
  • Training incentivizes guessing: The standard cross-entropy loss during pretraining, combined with evaluation metrics that penalize abstention (like "I don't know"), pushes models to guess rather than express uncertainty.
  • "Test-taking" mode: LLMs are optimized to be good "test-takers" on benchmarks that often use binary grading (correct/incorrect), making confident guesses more rewarding than admitting ignorance.
  • Socio-technical solution: The authors argue that simply adding more hallucination evaluations is insufficient. The core issue lies in modifying the primary evaluation benchmarks to reward uncertainty expression, which will then influence model development.
  • Calibration is key: The paper links miscalibration (overconfidence) to hallucinations and suggests that improving calibration is crucial for reducing them.

Key Results

  • The paper establishes a theoretical lower bound relating generative error rate (hallucinations) to IIV misclassification rate: err ≥ 2 * erriiv.
  • They show that even with error-free training data, the standard pretraining objective can lead to hallucinations.
  • The analysis suggests that the "singleton rate" (prompts appearing only once in training data) is a proxy for hallucination rates on arbitrary facts.
  • The paper provides theoretical bounds for hallucination rates in specific scenarios, such as arbitrary facts and multiple-choice questions.
  • An empirical analysis of common benchmarks (HELM, Open LLM Leaderboard, SWE-bench) reveals a widespread reliance on binary grading that penalizes abstention, reinforcing the problem.

Limitations and Future Work

  • Plausibility vs. Nonsense: The framework primarily considers "plausible falsehoods" and ignores the generation of nonsensical strings.
  • Open-ended Generations: While the framework can be adapted, the examples are mostly oriented towards factual questions, and degrees of hallucination in open-ended generation are not fully explored.
  • Search and Reasoning: While RAG and reasoning can reduce hallucinations, the underlying evaluation biases still reward guessing.
  • Latent Context: The current error definition does not account for "hidden context" external to the prompt.
  • False Trichotomy: The model doesn't distinguish between different magnitudes of errors or degrees of uncertainty beyond correct/incorrect/IDK.
  • Future Work: The authors advocate for modifying mainstream evaluations to explicitly state confidence targets and penalize guessing when uncertain. This socio-technical shift is seen as crucial for developing more trustworthy LLMs with richer pragmatic competence.

Summary updated

Abstract

Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. Such "hallucinations" persist even in state-of-the-art systems and undermine trust. We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and we analyze the statistical causes of hallucinations in the modern training pipeline. Hallucinations need not be mysterious -- they originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, then hallucinations in pretrained language models will arise through natural statistical pressures. We then argue that hallucinations persist due to the way most evaluations are graded -- language models are optimized to be good test-takers, and guessing when uncertain improves test performance. This "epidemic" of penalizing uncertain responses can only be addressed through a socio-technical mitigation: modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations. This change may steer the field toward more trustworthy AI systems.
BibTeX
@misc{kalai2025why,
  author = {Adam Tauman Kalai and Ofir Nachum and Santosh S. Vempala and Edwin Zhang},
  title = {{Why Language Models Hallucinate}},
  year = {2025},
  eprint = {2509.04664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.04664},
}
Figure