Language Models Hallucinate, but May Excel at Fact Verification

Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, Hao Peng

Abstract

Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently “hallucinate,” resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B, the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.

The Hallucination Challenge

Large Language Models (LLMs) are increasingly used for information retrieval and content creation due to their persuasive and fluent outputs. However, they are notorious for "hallucinating"—generating non-factual information that appears plausible. This lack of reliability is a major bottleneck for their deployment in high-stakes domains. This research investigates the extent of this problem and explores whether these same models can be repurposed to solve it by acting as automated fact verifiers.

Quantifying the Problem

To measure the severity of hallucinations, the researchers developed a new dataset of Model-Generated Statements (MGS). They evaluated outputs from four prominent LLMs: FLAN-T5, Llama-30B, Llama-65B, and GPT-3.5 across sentence completion (SENTCOM) and paragraph generation (PARAGEN) tasks.

Proportion of factual, unfactual, and no-evidence (NE) outputs across different models and tasks.

Human evaluation revealed a stark reality: even highly capable models like GPT-3.5 produce entirely factual paragraphs only 22.7% of the time when discussing Wikipedia topics. As shown in the data, as the complexity of the task increases from single sentences to full paragraphs, the proportion of factual content drops significantly across all models.

Methodology: LLMs as Fact Verifiers

The study proposes a systematic framework for LLM-based fact verification using several key techniques:

  • Retrieval Augmentation: Using Contriever to pull relevant passages from external corpora (like Wikipedia) to ground the model's judgment in evidence.
  • De-contextualization: Complex paragraphs are broken down into individual sentences. ChatGPT is used for coreference resolution (e.g., replacing pronouns with actual entity names) to ensure each sentence can be verified in isolation.
  • Soft Scoring: Instead of binary "Yes/No" answers, the system calculates a factuality score by normalizing the model's generation probabilities for specific tokens, allowing for more nuanced calibration.

The "Underdog" Winner: FLAN-T5

A primary finding of the study is that the ability to judge a fact is distinct from the ability to generate one. Surprisingly, FLAN-T5 (11B), which was the least factual generator in the study, emerged as the most effective fact verifier.

Pearson correlation of model-based fact verification with human judgments across various generation models.

As illustrated above, FLAN-T5 consistently achieved higher Pearson correlations with human judgments than much larger models like GPT-3.5 and ChatGPT. This suggests that instruction-tuned models like FLAN-T5 may be more robust at following verification prompts and less prone to the "distractions" or noise found in retrieved evidence that can lead larger models astray.

Key Insights and Robustness

The research highlighted several critical nuances in model performance:

  • Susceptibility to Noise: While FLAN-T5 is relatively stable, ChatGPT is more "distractible." When presented with a mix of relevant and irrelevant evidence, ChatGPT’s performance drops more significantly.
  • Numerical Weakness: All models struggle with "Cardinal" and "Ordinal" entities. They frequently misidentify factual statements involving numbers as hallucinations, likely due to difficulties in reasoning over inter-numeral logical relations.
  • Paragraph Strategy: The most effective way to verify long-form content is to aggregate scores from individual, de-contextualized sentences rather than attempting to verify the paragraph as a single unit.

Limitations and Future Work

While LLMs show great promise as verifiers, challenges remain. Most human-annotated data is currently limited to general knowledge (Wikipedia). Verifying highly technical fields like medicine or finance remains difficult for both general-purpose models and human annotators. Furthermore, the current single-step retrieval process could be improved by using autonomous reasoning agents capable of multi-step interactions to verify complex, multi-hop claims.

Summary updated

Abstract

Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently “hallucinate,” resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B , the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.
BibTeX
@inproceedings{guan2024language,
  author = {Jian Guan and Jesse Dodge and David Wadden and Minlie Huang and Hao Peng},
  title = {{Language Models Hallucinate, but May Excel at Fact Verification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=WVjWr4Kl9V},
}
Figure