Evaluating the Factual Consistency of Large Language Models Through News Summarization

Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, Colin Raffel

TL;DR

The Factual Inconsistency Benchmark (FIB) is a new evaluation framework designed to measure whether Large Language Models (LLMs) can distinguish between factually consistent and inconsistent summaries. Evaluating 23 LLMs across six model families reveals that while these models generally prefer factual consistency, they suffer from a severe "verbatim bias"—frequently choosing incorrect summaries simply because they copy text directly from the source document.

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at generating fluent text, they are notoriously prone to "hallucinations"—generating content that is factually incorrect or unsupported by the input text. While previous research heavily scrutinized factual consistency in smaller, supervised summarization models, less was known about how massive, zero-shot LLMs evaluate factuality.

This paper addresses this gap by investigating whether LLMs can inherently recognize factual consistency. Specifically, if we show an LLM a news article and two potential summaries—one accurate and one hallucinated—will it correctly assign a higher probability score to the true summary?

Methods

The authors introduce a paired-evaluation approach. For a given document, an LLM is presented with a factually consistent "Gold" summary and a factually inconsistent "Distractor" summary. The model's factual consistency is measured by its accuracy: the percentage of pairs where it assigns a higher score to the Gold summary.

A key contribution of the paper is the exploration of the scoring function. Standard log-likelihood (LL) is inherently biased toward shorter summaries and can be fooled if a summary contains highly common, generic phrases. To counteract this, the authors utilize length-normalized Pointwise Mutual Information (PMI) as their default scoring metric. This function subtracts the unconditional probability of the summary from its document-conditioned probability, effectively isolating how well the summary relates to the source text, and normalizes it by token length.

Datasets

The authors constructed the FIB (Factual Inconsistency Benchmark) dataset, which consists of over 3,500 manually annotated summary pairs built on top of two popular news datasets:

  • XSum: Used to evaluate abstractive summarization behavior.
  • CNN/DailyMail (CNN/DM): Used to evaluate extractive summarization behavior.

To ensure a high-quality ground truth, the authors manually verified and edited the reference summaries from these datasets to eliminate pre-existing hallucinations, creating a pristine "Gold" set. The factually inconsistent distractors were generated using a suite of 22 different summarization models and automatic perturbation techniques (such as entity swapping and sentence negation via FactCC).

Key Insights and Ideas

  • The Verbatim Trap: LLMs possess a massive blind spot for text copied verbatim from the input. If a distractor summary is composed of sentences lifted directly from the source document, LLMs overwhelmingly prefer it over a factually accurate, abstractive summary—even if the copied sentences are combined in a way that is completely false.
  • Instruction Tuning Backfire: Surprisingly, instruction-tuned models like T0 and FLAN-T5 performed worse at identifying factual consistency on XSum than their purely zero-shot counterparts. The authors hypothesize this is because these models were fine-tuned on standard summarization datasets that already contained noisy, factually inconsistent reference summaries, accidentally training the models to prefer hallucinations.
  • Scaling is a Slow Cure: While scaling model size (from 1B up to 176B parameters) generally improves performance within a model family, it does not completely eliminate the vulnerability to verbatim distractors or clever hallucinations.

Key Results

  • Abstractive Performance (XSum): When evaluating abstractive summaries, zero-shot LLMs perform reasonably well, achieving an accuracy of around 70% to 75% using the length-normalized PMI scoring function.
Figure 2: Performance of various models on FIB.
  • Extractive Vulnerability (CNN/DM): On the CNN/DM dataset, where distractors are model-extracted verbatim sentences, LLM accuracy plummets to roughly 10%. This highlights the models' overwhelming preference for verbatim text over factual accuracy.
  • Scoring Function Impact: Length-normalized PMI proved to be the most effective scoring function for zero-shot models, significantly outperforming standard log-likelihood, which struggled to differentiate between factual and non-factual summaries.
Figure 4: Performance of various models on FIB when using different scoring functions.

Limitations and Future Work

The primary limitation of FIB is its narrow domain focus; it evaluates factual consistency solely within the realm of English news summarization. It remains an open question whether these insights hold true for specialized domains (like medical or legal texts) or different tasks like open-domain question answering.

Additionally, the benchmark evaluates the models' scoring capabilities rather than their generation capabilities. Future work is needed to bridge this gap and develop decoding strategies that leverage metrics like PMI to actively prevent hallucinations during text generation.

Summary updated

Abstract

While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent. A model's factual consistency is then measured according to its accuracy, i.e.\ the proportion of documents where it assigns a higher score to the factually consistent summary. To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT. We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries. However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries. We validate design choices in our benchmark including the scoring method and source of distractor summaries. Our code and benchmark data can be found at https://github.com/r-three/fib.
BibTeX
@inproceedings{tam2023evaluating,
  author = {Derek Tam and Anisha Mascarenhas and Shiyue Zhang and Sarah Kwan and Mohit Bansal and Colin Raffel},
  title = {{Evaluating the Factual Consistency of Large Language Models Through News Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=UAXSIRNzXis},
}
Figure