The HALoGen Benchmark: Fantastic LLM Hallucinations and Where To Find Them

Abhilasha Ravichander, Shrusti Ghela, David Wadden, Yejin ChoiOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Despite their impressive ability to generate high-quality and fluent text, generative large language models (LLMs) also produce hallucinations: statements that are misaligned with established world knowledge or provided input context. How- ever, measuring hallucination can be challenging, as having humans verify model generations on-the-fly is both expensive and time-consuming. In this work, we re- lease HALOGEN , a comprehensive hallucination benchmark consisting of: (1) 10,923 prompts for generative models spanning nine domains including program- ming, scientific attribution, and summarization, and (2) automatic high-precision verifiers for each use case that decompose LLM generations into atomic units, and verify each unit against a high-quality knowledge source. We use this framework to evaluate ∼150,000 generations from 14 language models, finding that even the best-performing models . We further define a novel error classification for LLM hallucinations based on their source: (1) Type A errors for errors that may stem from incorrect recollection from training data, (2) Type B errors for errors that may stem from incorrect knowledge in training data or incorrect contextualization, and (3) Type C errors for hallucinations that are likely to be fabrication. For code packages, we that 70% of unique packages hallucinated by Llama-3-70B can be found in the C4 corpus, while for another category of hallucinations about fictional historic events, we find that we can seldom find a basis for these events within the data. We hope that our framework will provide a foundation to enable princi- pled scientific studies of why generative models hallucinate, and to advance the development of trustworthy large language models.
BibTeX
@misc{ravichander2025halogen,
  author = {Abhilasha Ravichander and Shrusti Ghela and David Wadden and Yejin Choi},
  title = {{The HALoGen Benchmark: Fantastic LLM Hallucinations and Where To Find Them}},
  year = {2025},
  url = {https://openreview.net/forum?id=pQ9QDzckB7},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure