Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output

Hithesh Sankararaman, Mohammed Nasheed Yasin, Tanner Sorensen, Alessandro Di Bari, Andreas StolckeEMNLP 2024

TL;DR

This paper introduces "Provenance," a lightweight and efficient fact-checking system for Large Language Model (LLM) generated text, specifically within Retrieval Augmented Generation (RAG) pipelines. By leveraging compact Natural Language Inference (NLI) models instead of heavier LLMs, Provenance achieves competitive accuracy in detecting nonfactual claims with significantly lower latency and cost, making LLM fact-checking more accessible.

What Problem Does This Paper Address?

As LLMs become increasingly integrated into applications for tasks like question answering and summarization, ensuring the factual accuracy of their outputs is paramount. LLMs, however, are prone to "hallucinations" – generating plausible-sounding but incorrect information. This can lead to misinformation and erode trust in AI systems. Existing fact-checking methods often rely on other LLMs, which can be slow, expensive, and also susceptible to hallucinations. The paper aims to bridge this gap by providing a more efficient and reliable fact-checking mechanism for RAG-generated content.

Methods

Provenance employs a two-stage approach using specialized cross-encoder models:

  1. Relevancy Scorer: This component takes the original query and the retrieved context items (provided by the RAG system) and scores each context item's relevance to the query. This helps in filtering out irrelevant information.
  2. Context Item Selection: Based on the relevance scores, a subset of the most relevant context items (termed "sources") is selected. The paper explores two strategies for this selection: TopK (similar to retrieval ranking) and TopP (nucleus sampling). The relevance scores of these selected sources are then normalized to act as weights.
  3. Fact Checker: This is the core NLI component. It takes the user's query, the generated answer, and the selected context sources. It constructs a "claim prompt" (e.g., "The answer to question X is Y") and pairs it with each source. A specialized NLI cross-encoder model then scores how well each source supports the claim. These scores are aggregated using the source weights to produce a final factuality score for the LLM's output. This score can be thresholded to classify the output as factual or nonfactual.

The key innovation here is the use of compact, pre-trained NLI cross-encoders, which are inherently less prone to hallucination than generative LLMs and offer much faster inference.

Datasets

Provenance was evaluated on a diverse set of open-source datasets, including:

  • TRUE: A comprehensive dataset designed to test factual accuracy across various domains and question types.
  • MSMarco: A large-scale dataset for machine reading comprehension, useful for evaluating web-based information retrieval.
  • TruthfulQA: Specifically designed to test LLM truthfulness and their ability to handle deceptive questions.
  • HotpotQA: A multi-hop question-answering dataset requiring reasoning over multiple evidence pieces.
  • HaluEval: A benchmark for evaluating LLMs' ability to recognize hallucinated samples.
  • HaluBench: An open-source benchmark focusing on hallucination tasks sourced from real-world domains like finance and medicine.

The paper also details data preparation steps, including splitting sources into sentences and generating hallucinated answers for datasets that lacked them to ensure balanced evaluation.

Key Insights and Ideas

  • Lightweight NLI for Fact-Checking: The core idea that compact NLI models can effectively and efficiently fact-check LLM outputs is a significant contribution. This moves away from the "LLM judging LLM" paradigm, which can be a bottleneck.
  • Relevance Scoring is Crucial: The paper emphasizes that simply passing all retrieved context to a fact-checker is suboptimal. Filtering and weighting context based on relevance to the query significantly improves performance.
  • TopP Selection Strategy: The TopP selection strategy for context items showed promising results, suggesting that a more nuanced approach to context filtering can be beneficial.
  • Interpretable Judgments: The use of cross-encoders, while not fully interpretable in the way human explanations are, offers more transparency than black-box LLM judgments.

Key Results

  • Provenance achieved competitive hallucination detection performance, measured by Area Under the ROC Curve (AUC), across a wide range of datasets.
  • On the TRUE dataset, Provenance achieved the best AUC for 3 out of 7 datasets (DialFact, MNBM, and Q2).
  • In comparisons with other methods, Provenance demonstrated strong performance, often surpassing or being competitive with much larger LLM-based fact-checkers, despite its significantly smaller model size (around 300M parameters compared to LLMs in the tens or hundreds of billions).
  • For example, on the summarization task in HaluEval, Provenance achieved the best accuracy, surpassing ChatGPT by 3.74%.
  • On the HaluBench, Provenance surpassed GPT-3.5-Turbo by 3.38% and was only 3.32% behind Claude-3-Haiku, despite the latter having significantly more parameters.

Limitations and Future Work

  • Context Chunking: The paper acknowledges that reliable semantic chunking of contexts remains an evolving field and that their approach to chunking generated text is still an area for further research.
  • Scope of Scenarios: The current focus is primarily on text sources and short context/short answer scenarios, with some exploration into longer contexts. Future work could expand to other data modalities (conversations, tables).
  • Systematic Optimization: The authors mention not carrying out systematic optimization of TopP and TopK values, suggesting potential for further tuning.
  • LLM Ablation/Introspection: While Provenance is presented as an alternative, the paper acknowledges existing LLM-based methods (ablation and introspection) and doesn't directly compare against all of them in every experiment.

Future work could involve exploring more advanced context chunking strategies, expanding to multi-modal RAG systems, and further optimizing the selection and scoring mechanisms.

Summary updated

BibTeX
@inproceedings{sankararaman2024provenance,
  author = {Hithesh Sankararaman and Mohammed Nasheed Yasin and Tanner Sorensen and Alessandro Di Bari and Andreas Stolcke},
  title = {{Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure