Annotating and Modeling Fine-grained Factuality in Summarization

Tanya Goyal, Greg Durrett

Abstract

Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing. We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSum. Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data. Finally, we show that our best factuality detection model enables training of more factual XSum summarization models by allowing us to identify non-factual tokens in the training data.

Introduction

Abstractive summarization—where an AI generates a summary in its own words—is prone to "hallucinations," where the model produces text that sounds plausible but is factually unsupported by the source document. Current efforts to build "factuality detectors" often rely on synthetic data, such as taking a correct sentence and intentionally swapping a name or adding a negation. However, there has been a significant gap in understanding whether these artificial errors reflect the complex failures of modern models like BART or PEGASUS. This paper investigates whether a "fine-grained" approach—analyzing specific word-to-word dependencies—can more effectively identify and prevent these errors.

Methods and Datasets

The researchers compared several modeling approaches, centered on Dependency Arc Entailment (DAE). Rather than judging an entire sentence as a single unit, the DAE model decomposes a summary into its constituent "dependency arcs"—the grammatical relationships between words (e.g., subject → verb). The model then evaluates whether each specific relationship is supported by the source text. To handle cases where only sentence-level labels are available, they introduced DAE-Weak, a training variant that uses a loss function assuming at least one arc is incorrect if the sentence is labeled non-factual.

The study utilized two distinct summarization domains:

  • CNN/DailyMail: A dataset characterized by "extractive" summaries that stay close to the original text.
  • XSum (Extreme Summarization): A highly abstractive dataset where models must synthesize information, leading to significantly higher hallucination rates.

To train these models, the authors contrasted synthetic datasets—Ent-C (entity-centric swaps) and Gen-C (paraphrase-based errors)—with a new Human-XSUM dataset containing human-annotated labels at the word and span level.

Comparison of synthetic training data (Ent-C, Gen-C) versus human annotation, highlighting non-factual spans and dependency arcs.

Key Insights: The Synthetic Mismatch

A critical finding is that synthetic datasets are often "too easy." Models trained on them learn to spot simple name swaps but remain blind to the nuanced "intrinsic" errors real models make, such as incorrectly linking two facts that both appear in the source but are unrelated. Furthermore, the authors discovered a startling reality: 75% of the "gold" human-written summaries in the XSum dataset contain factual errors. This implies that summarization models are frequently being trained to hallucinate.

Results and Impact

Fine-grained human annotations proved to be a far superior training signal. A DAE model trained on just 2,000 human-labeled examples achieved 78.7% accuracy on the challenging XSum dataset, compared to just ~51-54% for models trained on tens of thousands of synthetic examples.

Beyond detection, the authors demonstrated how to improve summarization itself. By using their factuality model to "mask" non-factual tokens during the training of a new XSum summarizer, they effectively told the model: "Predict the rest of the summary, but do not learn from these specific broken words." This approach reduced the Word Error Rate (Word-ER) from 32.2% to 23.7%. In human evaluations, summaries generated with this factuality-masking were rated significantly higher (46.5 vs 37.4) than standard baselines.

Conclusion

The study highlights that factuality detection is highly domain-dependent; models trained on one dataset (like CNN/DM) do not transfer well to others (like XSum). While fine-grained human annotation is resource-intensive, it is essential for developing reliable fact-checkers. Future research should focus on creating more realistic synthetic data and investigating how to incorporate fine-grained detectors directly into the decoding process to prevent hallucinations in real-time.

Summary updated

Abstract

Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing. We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSum. Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data. Finally, we show that our best factuality detection model enables training of more factual XSum summarization models by allowing us to identify non-factual tokens in the training data.
BibTeX
@inproceedings{goyal2021annotating,
  author = {Tanya Goyal and Greg Durrett},
  title = {{Annotating and Modeling Fine-grained Factuality in Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=zxebmsfx6Am},
}
Figure