The HALoGen Benchmark: Fantastic LLM Hallucinations and Where To Find Them
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Despite their impressive ability to generate high-quality and fluent text, generative large language models (LLMs) also produce hallucinations: statements that are misaligned with established world knowledge or provided input context. How- ever, measuring hallucination can be challenging, as having humans verify model generations on-the-fly is both expensive and time-consuming. In this work, we re- lease HALOGEN , a comprehensive hallucination benchmark consisting of: (1) 10,923 prompts for generative models spanning nine domains including program- ming, scientific attribution, and summarization, and (2) automatic high-precision verifiers for each use case that decompose LLM generations into atomic units, and verify each unit against a high-quality knowledge source. We use this framework to evaluate ∼150,000 generations from 14 language models, finding that even the best-performing models . We further define a novel error classification for LLM hallucinations based on their source: (1) Type A errors for errors that may stem from incorrect recollection from training data, (2) Type B errors for errors that may stem from incorrect knowledge in training data or incorrect contextualization, and (3) Type C errors for hallucinations that are likely to be fabrication. For code packages, we that 70% of unique packages hallucinated by Llama-3-70B can be found in the C4 corpus, while for another category of hallucinations about fictional historic events, we find that we can seldom find a basis for these events within the data. We hope that our framework will provide a foundation to enable princi- pled scientific studies of why generative models hallucinate, and to advance the development of trustworthy large language models.
BibTeX
@misc{ravichander2025halogen,
author = {Abhilasha Ravichander and Shrusti Ghela and David Wadden and Yejin Choi},
title = {{The HALoGen Benchmark: Fantastic LLM Hallucinations and Where To Find Them}},
year = {2025},
url = {https://openreview.net/forum?id=pQ9QDzckB7},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}