TL;DR
The Factual Inconsistency Benchmark (FIB) is a new evaluation framework designed to measure whether Large Language Models (LLMs) can distinguish between factually consistent and inconsistent summaries. Evaluating 23 LLMs across six model families reveals that while these models generally prefer factual consistency, they suffer from a severe "verbatim bias"—frequently choosing incorrect summaries simply because they copy text directly from the source document.
What Problem Does This Paper Address?
While Large Language Models (LLMs) excel at generating fluent text, they are notoriously prone to "hallucinations"—generating content that is factually incorrect or unsupported by the input text. While previous research heavily scrutinized factual consistency in smaller, supervised summarization models, less was known about how massive, zero-shot LLMs evaluate factuality.
This paper addresses this gap by investigating whether LLMs can inherently recognize factual consistency. Specifically, if we show an LLM a news article and two potential summaries—one accurate and one hallucinated—will it correctly assign a higher probability score to the true summary?
Methods
The authors introduce a paired-evaluation approach. For a given document, an LLM is presented with a factually consistent "Gold" summary and a factually inconsistent "Distractor" summary. The model's factual consistency is measured by its accuracy: the percentage of pairs where it assigns a higher score to the Gold summary.
A key contribution of the paper is the exploration of the scoring function. Standard log-likelihood (LL) is inherently biased toward shorter summaries and can be fooled if a summary contains highly common, generic phrases. To counteract this, the authors utilize length-normalized Pointwise Mutual Information (PMI) as their default scoring metric. This function subtracts the unconditional probability of the summary from its document-conditioned probability, effectively isolating how well the summary relates to the source text, and normalizes it by token length.
Datasets
The authors constructed the FIB (Factual Inconsistency Benchmark) dataset, which consists of over 3,500 manually annotated summary pairs built on top of two popular news datasets:
- XSum: Used to evaluate abstractive summarization behavior.
- CNN/DailyMail (CNN/DM): Used to evaluate extractive summarization behavior.
To ensure a high-quality ground truth, the authors manually verified and edited the reference summaries from these datasets to eliminate pre-existing hallucinations, creating a pristine "Gold" set. The factually inconsistent distractors were generated using a suite of 22 different summarization models and automatic perturbation techniques (such as entity swapping and sentence negation via FactCC).
Key Insights and Ideas
- The Verbatim Trap: LLMs possess a massive blind spot for text copied verbatim from the input. If a distractor summary is composed of sentences lifted directly from the source document, LLMs overwhelmingly prefer it over a factually accurate, abstractive summary—even if the copied sentences are combined in a way that is completely false.
- Instruction Tuning Backfire: Surprisingly, instruction-tuned models like T0 and FLAN-T5 performed worse at identifying factual consistency on XSum than their purely zero-shot counterparts. The authors hypothesize this is because these models were fine-tuned on standard summarization datasets that already contained noisy, factually inconsistent reference summaries, accidentally training the models to prefer hallucinations.
- Scaling is a Slow Cure: While scaling model size (from 1B up to 176B parameters) generally improves performance within a model family, it does not completely eliminate the vulnerability to verbatim distractors or clever hallucinations.
Key Results
- Abstractive Performance (XSum): When evaluating abstractive summaries, zero-shot LLMs perform reasonably well, achieving an accuracy of around 70% to 75% using the length-normalized PMI scoring function.
- Extractive Vulnerability (CNN/DM): On the CNN/DM dataset, where distractors are model-extracted verbatim sentences, LLM accuracy plummets to roughly 10%. This highlights the models' overwhelming preference for verbatim text over factual accuracy.
- Scoring Function Impact: Length-normalized PMI proved to be the most effective scoring function for zero-shot models, significantly outperforming standard log-likelihood, which struggled to differentiate between factual and non-factual summaries.
Limitations and Future Work
The primary limitation of FIB is its narrow domain focus; it evaluates factual consistency solely within the realm of English news summarization. It remains an open question whether these insights hold true for specialized domains (like medical or legal texts) or different tasks like open-domain question answering.
Additionally, the benchmark evaluates the models' scoring capabilities rather than their generation capabilities. Future work is needed to bridge this gap and develop decoding strategies that leverage metrics like PMI to actively prevent hallucinations during text generation.
Summary updated