Improving Truthfulness of Headline Generation

Kazuki Matsumaru, Sho Takase, Naoaki Okazaki

Abstract

Most studies on abstractive summarization report ROUGE scores between system and reference summaries. However, we have a concern about the truthfulness of generated summaries: whether all facts of a generated summary are mentioned in the source text. This paper explores improving the truthfulness in headline generation on two popular datasets. Analyzing headlines generated by the state-of-the-art encoder-decoder model, we show that the model sometimes generates untruthful headlines. We conjecture that one of the reasons lies in untruthful supervision data used for training the model. In order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article entails its headline. After confirming quite a few untruthful instances in the datasets, this study hypothesizes that removing untruthful instances from the supervision data may remedy the problem of the untruthful behaviors of the model. Building a binary classifier that predicts an entailment relation between an article and its headline, we filter out untruthful instances from the supervision data. Experimental results demonstrate that the headline generation model trained on filtered supervision data shows no clear difference in ROUGE scores but remarkable improvements in automatic and manual evaluations of the generated headlines.

The Problem: Hallucinations in Abstractive Summarization

In the field of Natural Language Processing (NLP), abstractive summarization is often considered the ultimate goal—rewriting text into a concise version using novel vocabulary. However, these models have a notorious tendency to "hallucinate," including specific facts like dates, locations, or names that are not present in the source document.

The authors argue that this is not merely a failure of model architecture, but a fundamental data problem. The standard benchmark for headline generation, the English Gigaword dataset, typically pairs the first sentence of a news article with its headline. However, headlines are often written by editors who have access to the entire story. Consequently, a headline might mention specific details like "November" or "The Detroit News" that do not appear in the lead sentence. Training models on this noisy data effectively teaches them to guess or fabricate details to satisfy the training objective.

Examples of untruthful headlines generated by models trained on noisy data.

The ROUGE Paradox

A striking finding of this research is that ROUGE scores—the standard metric for summarization performance—are almost entirely blind to factual truthfulness. The study found a very low correlation (Pearson’s $r = 0.189$) between ROUGE-1 and the "Support Score" (a metric for truthfulness). This means a model can achieve a high ROUGE score by matching the reference headline's style while still hallucinating incorrect facts.

Scatter plot showing the low correlation between ROUGE-1 scores and Support scores (truthfulness).

Methodology: Filtering and Self-Training

To address the issue of "untruthful supervision," the researchers proposed a three-step pipeline:

  1. Quantifying Truthfulness: They introduced a "Support Score," essentially a recall-oriented ROUGE-1 score between the source text and the generated headline. If the headline contains words not found in the source, the score drops.
  2. Entailment Filtering: The problem was treated as a Textual Entailment task. Using RoBERTa-large (English) and BERT (Japanese), they built binary classifiers to determine if a source sentence logically entails its headline. They found that roughly 30% of Gigaword pairs and 40% of Japanese pairs failed this test and filtered them out.
  3. Self-Training (Pseudo-Labeling): To maintain training volume after filtering, they used a model trained on the clean data to generate new, truthful headlines for the discarded source sentences, creating a large, synthetic, but factually grounded dataset.

Key Results and Insights

The "Filtered + Pseudo" approach yielded significant improvements in factual accuracy. On the Gigaword dataset, the ratio of truthful headlines (verified by human evaluation) jumped from 77.06% to 85.32%. Crucially, these gains in truthfulness came with no significant drop in ROUGE scores; in fact, ROUGE-1 on Gigaword slightly improved from 35.80 to 35.85.

The study also revealed that the common practice of using only the first sentence of an article (the "Lead-1" heuristic) is a major source of noise. When the authors used the full article text, the entailment ratio jumped from ~70% to ~93%, proving the information was usually present in the article, just not in the sentence provided to the model during standard training.

Conclusion and Future Work

This research demonstrates that cleaning training data through entailment filtering can significantly mitigate hallucination in summarization models. While the "Support Score" is a helpful proxy, future work is needed to develop automated metrics that better handle paraphrasing and semantic truth. Additionally, moving beyond binary filtering to more granular editing of headlines could further refine the quality of supervision data in domains where factual accuracy is paramount, such as medical or legal summarization.

Summary updated

Abstract

Most studies on abstractive summarization report ROUGE scores between system and reference summaries. However, we have a concern about the truthfulness of generated summaries: whether all facts of a generated summary are mentioned in the source text. This paper explores improving the truthfulness in headline generation on two popular datasets. Analyzing headlines generated by the state-of-the-art encoder-decoder model, we show that the model sometimes generates untruthful headlines. We conjecture that one of the reasons lies in untruthful supervision data used for training the model. In order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article entails its headline. After confirming quite a few untruthful instances in the datasets, this study hypothesizes that removing untruthful instances from the supervision data may remedy the problem of the untruthful behaviors of the model. Building a binary classifier that predicts an entailment relation between an article and its headline, we filter out untruthful instances from the supervision data. Experimental results demonstrate that the headline generation model trained on filtered supervision data shows no clear difference in ROUGE scores but remarkable improvements in automatic and manual evaluations of the generated headlines.
BibTeX
@inproceedings{matsumaru2020improving,
  author = {Kazuki Matsumaru and Sho Takase and Naoaki Okazaki},
  title = {{Improving Truthfulness of Headline Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=yOka2itYZBk},
}
Figure