Abstract
While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB (Factual Inconsistency Benchmark) that focuses on the task of summarization. Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent. A model's factual consistency is then measured according to its accuracy, i.e., the proportion of documents where it assigns a higher score to the factually consistent summary. To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT. We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries. However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries. We validate design choices in our benchmark including the scoring method and source of distractor summaries.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their tendency to "hallucinate"—generating text that is fluent but factually incorrect—remains a significant barrier to their deployment in high-stakes environments. While previous research has focused on improving the generation process, a more fundamental question remains: Can LLMs distinguish between factual truth and plausible-sounding fiction?
Researchers at UNC Chapel Hill address this by introducing the Factual Inconsistency Benchmark (FIB). Unlike standard metrics that evaluate the quality of model-generated text, FIB measures an LLM's internal preference for factual consistency by asking it to score pre-written summaries of news articles.
Methodology: The FIB Framework
The FIB benchmark frames factual consistency as a binary classification task. For any given news article, a model is presented with two options:
- The Gold Summary: A human-written summary manually verified to be factually consistent with the source text.
- The Distractor Summary: A summary generated by an AI model that has been manually annotated as factually inconsistent.
Scoring with PMI
A critical contribution of this work is the scoring mechanism. Standard log-likelihood (LL) is often biased toward shorter sequences or generic, high-probability phrases. To isolate the model's understanding of the document's content, the researchers utilize length-normalized Pointwise Mutual Information (PMI).
PMI calculates the probability of the summary given the document, while subtracting the probability of the summary appearing in isolation. This ensures the score reflects how much the source document specifically supports the summary, rather than how "likely" the summary text is on its own.
Dataset Composition
The FIB benchmark consists of over 3,500 summary pairs derived from two primary news datasets:
- XSum: Focused on abstractive summarization, where summaries use novel wording not found in the source.
- CNN/Daily Mail (CNN/DM): Focused on extractive summarization, where summaries often reuse phrases directly from the text.
The distractors were sourced from 22 different summarization models and automated tools designed to introduce specific errors, such as swapping entities (names, dates, or locations).
Key Insights
The Extractive Bias
The most significant finding is the "extractive bias." LLMs show a strong preference for text that appears verbatim in the source document. On the CNN/DM dataset, models frequently assigned higher scores to factually incorrect summaries simply because they were copied word-for-word from the article. This suggests that LLMs may prioritize surface-level lexical overlap over actual semantic consistency.
The Instruction-Tuning Paradox
Interestingly, models fine-tuned to follow instructions (such as T0 and FLAN-T5) often performed worse on the FIB benchmark than their base counterparts (like GPT-3 or BLOOM). The researchers hypothesize this is due to the "noisy" nature of instruction-tuning datasets, which often include reference summaries that are not perfectly factually consistent with their source texts.
Scaling and Performance
While model scale generally improves performance—with 176B parameter models like BLOOM and OPT-175B leading the field—no model achieved perfect accuracy. On the XSum dataset, top models reached approximately 72-75% accuracy. This indicates that even the largest models currently available struggle to reliably identify hallucinations.
Conclusion
The FIB benchmark provides a rigorous framework for evaluating the "truth preference" of LLMs. The results highlight a critical vulnerability: the extractive bias. As LLMs continue to evolve, moving beyond surface-level fluency to a deeper, more robust understanding of factual consistency remains a primary challenge. The FIB benchmark serves as a necessary reality check, reminding us that for current AI, "sounding right" is not the same as "being right."
For more details, the code and benchmark data are available at https://github.com/r-three/fib.
Summary updated