Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis

Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi, Karim Ghonim, Federico Martelli, Roberto NavigliarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) frequently generate "hallucinations"—content that is fluent and authoritative yet factually incorrect. This phenomenon poses a significant barrier to the safe and reliable deployment of AI in critical domains. While existing benchmarks address factuality, they often focus on isolated claim verification or specific domains, failing to capture the complexity of real-world, long-form text. To address this, we introduce LLM-OASIS, a massive dataset designed for end-to-end factuality evaluation.

Methodology: The LLM-OASIS Pipeline

The core contribution of this work is an automated four-step pipeline designed to generate high-quality factual and unfactual text pairs from Wikipedia.

  1. Claim Extraction: Using GPT-4, the system extracts atomic, self-contained claims from a source Wikipedia passage.
  2. Claim Falsification: The model subtly alters one specific claim to introduce a critical factual inaccuracy that remains contextually plausible (e.g., swapping geographically related entities).
  3. Factual Text Generation: The original passage is paraphrased to create a clean, factual baseline.
  4. Unfactual Text Generation: A nearly identical version of the factual text is generated, incorporating the single falsified claim.
The LLM-OASIS data creation pipeline illustrating the transition from claim extraction to unfactual text generation.

To evaluate these texts, the authors propose a modular architecture that extracts claims from target text, retrieves evidence from a 2.5-million-passage Wikipedia knowledge base, and verifies claims using Natural Language Inference (NLI).

Dataset Characteristics

LLM-OASIS consists of two primary components: a large-scale "Silver" training set and a "Gold" standard benchmark. The Silver dataset contains over 81,000 text pairs and over 680,000 associated claims, providing a rich resource for training specialized models. The Gold standard is a manually curated subset of approximately 2,000 instances, verified by expert linguists to ensure absolute ground truth.

Detailed statistics for the LLM-OASIS dataset across its various generation stages.

The dataset is characterized by "subtle mirages"—falsifications that use domain-specific substitutions rather than simple negations. This makes detection significantly more challenging for current models as the resulting text remains highly coherent.

Key Results and Insights

The evaluation of state-of-the-art models on LLM-OASIS yielded several critical insights:

  • Modular Superiority: A pipeline of smaller, specialized models (totaling ~1B parameters) outperformed massive general-purpose models like GPT-4o. The specialized system achieved 69.24% accuracy on the end-to-end task, compared to GPT-4o's 68.0% (with retrieval).
  • RAG Challenges: Interestingly, Retrieval-Augmented Generation (RAG) sometimes hindered performance for open-weight models. These models often struggled to reason over retrieved evidence, suggesting that factuality failures are often rooted in reasoning deficits rather than simple memory gaps.
  • Performance Benchmarks: GPT-4o achieved 60.8% accuracy in a zero-shot setting. Llama-3.3-70B reached 61.7% with few-shot prompting. On the simpler task of evidence-based claim verification, the specialized 0.4B parameter model reached an impressive 93.3% accuracy.

Conclusion and Future Work

LLM-OASIS provides a rigorous new benchmark for the AI community to measure and improve the factuality of LLMs. While currently focused on English Wikipedia, future iterations will expand to include scientific literature, news domains, and multiple languages. Addressing the temporal sensitivity of factual claims—where truths change as Wikipedia is updated—remains a key area for ongoing research.

Summary updated

Abstract

After the introduction of Large Language Models (LLMs), there have been substantial improvements in the performance of Natural Language Generation (NLG) tasks, including Text Summarization and Machine Translation. However, LLMs still produce outputs containing hallucinations, that is, content not grounded in factual information. Therefore, developing methods to assess the factuality of LLMs has become urgent. Indeed, resources for factuality evaluation have recently emerged. Although challenging, these resources face one or more of the following limitations: (i) they are tailored to a specific task or domain; (ii) they are limited in size, thereby preventing the training of new factuality evaluators; (iii) they are designed for simpler verification tasks, such as claim verification. To address these issues, we introduce LLM-Oasis, to the best of our knowledge the largest resource for training end-to-end factuality evaluators. LLM-Oasis is constructed by extracting claims from Wikipedia, falsifying a subset of these claims, and generating pairs of factual and unfactual texts. We then rely on human annotators to both validate the quality of our dataset and to create a gold standard test set for benchmarking factuality evaluation systems. Our experiments demonstrate that LLM-Oasis presents a significant challenge for state-of-the-art LLMs, with GPT-4o achieving up to 60% accuracy in our proposed end-to-end factuality evaluation task, highlighting its potential to drive future research in the field.
BibTeX
@misc{scire2024truth,
  author = {Alessandro Scirè and Andrei Stefan Bejgu and Simone Tedeschi and Karim Ghonim and Federico Martelli and Roberto Navigli},
  title = {{Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis}},
  year = {2024},
  eprint = {2411.19655},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19655},
}
Figure