The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs

Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, Tomasz Jan Kajdanowicz

TL;DR

This paper reveals that common metrics like ROUGE significantly overestimate the performance of hallucination detection methods in Large Language Models (LLMs). By comparing these metrics against human judgments and LLM-as-Judge evaluations, the authors demonstrate that simple heuristics like response length can be surprisingly effective, highlighting a fundamental flaw in current evaluation practices and the need for more robust, semantically aware metrics.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly deployed in critical applications, but their tendency to "hallucinate" – generate fluent but factually incorrect outputs – poses a significant risk. Detecting these hallucinations is crucial for reliable deployment. However, existing methods for evaluating hallucination detection often rely on metrics like ROUGE, which are based on lexical overlap. The authors argue that ROUGE, and similar metrics, are poorly aligned with human judgments of factual correctness, leading to an "illusion of progress" where methods appear more effective than they truly are. This gap makes it difficult to accurately assess the trustworthiness of LLM outputs.

Methods

The authors conducted a multi-pronged investigation:

  • Human Evaluation Study: They collected human judgments on factual correctness for LLM-generated answers.
  • LLM-as-Judge Evaluation: They employed GPT-4o-Mini as an automated evaluator, which has shown strong agreement with human judgments, to assess factual correctness.
  • Re-evaluation of Existing Methods: They re-evaluated several established unsupervised hallucination detection methods (e.g., Perplexity, Semantic Entropy, Eigenscore) using both ROUGE and LLM-as-Judge metrics.
  • Controlled Experiments: They designed experiments to isolate the effect of response length and input ambiguity on hallucination rates and evaluation metric performance.
  • Analysis of Alternative Metrics: They compared ROUGE and LLM-as-Judge against other lexical (BLEU) and semantic (BERTScore, UniEval) metrics.

Datasets

The study utilized three established Question Answering (QA) datasets:

  • NQ-Open: 3,610 question-answer pairs from real Google search queries.
  • TriviaQA: 3,842 examples from the validation set, featuring trivia questions.
  • SQuAD (v2): 4,150 examples from the validation set, characterized by longer questions and answers.

The authors also referenced the HaluEval dataset for analysis of summarization and dialogue tasks.

Key Insights and Ideas

  • ROUGE is Misleading: ROUGE's reliance on lexical overlap leads to a significant overestimation of hallucination detection performance, particularly due to its low precision. It often fails to recognize semantic equivalence and is easily manipulated by response length.
  • LLM-as-Judge as a Reliable Proxy: LLM-as-Judge evaluations, when validated against human judgments, provide a much more accurate assessment of factual correctness and align better with human intuition.
  • Length is a Powerful (and Problematic) Signal: Response length is a surprisingly strong indicator of hallucinations. Longer answers tend to be associated with incorrectness, and metrics like ROUGE are biased against longer responses, even if factually correct. This suggests that some "sophisticated" detectors might be inadvertently capturing length rather than true semantic errors.
  • Simple Baselines Can Compete: Basic heuristics based on response length (e.g., average length across multiple generations) can achieve performance comparable to, or even exceeding, more complex detection methods.
  • Input Ambiguity is a Major Trigger: Ambiguous questions significantly increase response length and hallucination rates, suggesting that input formulation is a critical factor.

Key Results

  • Several established hallucination detection methods showed performance drops of up to 45.9% (Perplexity) and 30.4% (Eigenscore) when re-evaluated using LLM-as-Judge compared to ROUGE.
  • ROUGE exhibits high recall but alarmingly low precision, leading to a significant misalignment with human judgments. LLM-as-Judge achieved high precision and recall, aligning closely with human assessments.
  • The correlation between ROUGE AUROC scores and LLM-as-Judge AUROC scores was moderate (r = 0.55), indicating that methods optimized for ROUGE might not generalize to factual correctness.
  • Few-shot prompting improved evaluation reliability by standardizing response formats, but did not eliminate evaluation biases.
  • Alternative metrics like BLEU, BERTScore, and UniEval-fact also showed limited alignment with human judgments when evaluated against LLM-as-Judge.

Limitations and Future Work

  • Scope: The study focused on a subset of LLMs and datasets, and its generalizability to other models and tasks needs further validation.
  • Length Heuristic Nuances: While effective, response length might not account for all cases where longer responses are factually accurate.
  • LLM-as-Judge Biases: While more robust than ROUGE, LLM-as-Judge is not without its own biases and limitations.
  • Future Work: The authors suggest expanding the analysis to a broader range of models and tasks, developing metrics that are robust against manipulation (like length-based biases), and further investigating the causal mechanisms behind hallucination. They also highlight the need for more semantically aware evaluation frameworks to ensure the trustworthiness of LLM outputs.

Summary updated

Abstract

Large language models (LLMs) have revolutionized natural language processing, yet their tendency to hallucinate poses serious challenges for reliable deployment. Despite numerous hallucination detection methods, their evaluations often rely on ROUGE, a metric based on lexical overlap that misaligns with human judgments. Through comprehensive human studies, we demonstrate that while ROUGE exhibits high recall, its extremely low precision leads to misleading performance estimates. In fact, several established detection methods show performance drops of up to 45.9\% when assessed using human-aligned metrics like LLM-as-Judge. Moreover, our analysis reveals that simple heuristics based on response length can rival complex detection techniques, exposing a fundamental flaw in current evaluation practices. We argue that adopting semantically aware and robust evaluation frameworks is essential to accurately gauge the true performance of hallucination detection methods, ultimately ensuring the trustworthiness of LLM outputs.
BibTeX
@inproceedings{janiak2025illusion,
  author = {Denis Janiak and Jakub Binkowski and Albert Sawczyn and Bogdan Gabrys and Ravid Shwartz-Ziv and Tomasz Jan Kajdanowicz},
  title = {{The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure