Automatically Discarding Straplines to Improve Data Quality for Abstractive News Summarization

Amr Keleg, Matthias Lindemann, Danyang Liu, Wanqiu Long, Bonnie L. WebberACL 2022

TL;DR

This paper tackles the issue of "straplines" – short, attention-grabbing phrases often mistaken for summaries – in automatically generated news datasets. By developing a rule-based system to identify and remove these straplines, the authors demonstrate a significant improvement in the quality of abstractive news summaries, leading to better performance in downstream summarization models.

What Problem Does This Paper Address?

Modern abstractive news summarization models heavily rely on large datasets of news articles paired with their summaries. These datasets are often created by scraping websites, a process that can inadvertently include non-summary texts. One such problematic text type is the "strapline," which, while intended to pique reader interest, often doesn't function as a true summary. The presence of these straplines in training data can degrade the quality of summarization models, leading them to generate less informative or even nonsensical summaries, and can also render evaluation metrics less meaningful if the reference "summary" is actually a strapline. This paper aims to address this data quality issue by identifying and removing straplines from existing datasets.

Methods

The authors propose a two-pronged approach: first, a noise filtration step, and second, a strapline detection pipeline.

  • Noise Filtration: This initial step removes various forms of unwanted text, including remnants of web formatting, truncated sentences, datelines, overly short extracts, and non-English content.
  • Strapline Detection Pipeline: This pipeline employs a set of rule-based heuristics designed to identify straplines. These heuristics are based on observed characteristics of straplines, such as:
  • Beginning with imperative speech (e.g., "Check out...").
  • High quote coverage.
  • Use of first or second-person pronouns.
  • Use of question or exclamation marks.
  • Being a repeated extract across multiple articles.
  • Being classified as "clickbait" by a fine-tuned BERT model.

These heuristics are combined to classify extracts as straplines or not.

Datasets

  • Newsroom Corpus: This is the primary dataset used in the study. The authors focus on extracts from the New York Times (NYT) and Time.com sections of this corpus.
  • Webis-Clickbait-17 Dataset: Used to fine-tune a BERT model for clickbait detection, which is then incorporated into the strapline detection heuristics.

The authors also created manually annotated development and test sets for evaluating their strapline detection method.

Key Insights and Ideas

  • Straplines are a significant data quality issue: The study reveals that a substantial portion of extracts in scraped datasets, like Newsroom, are actually straplines, not summaries. This is particularly true for Time.com.
  • Straplines have distinct characteristics: The paper successfully identifies and operationalizes several linguistic and structural features that differentiate straplines from summaries.
  • Removing straplines improves summarization quality: The core finding is that cleaning the training data by removing straplines leads to demonstrably better abstractive summaries.
  • Impact varies by source: While removing straplines benefits both NYT and Time.com, the improvement is more pronounced for NYT, potentially due to issues with summary repetition in the original NYT training data.
  • Qualitative improvements are key: While ROUGE scores show improvements, the paper highlights that qualitative differences (e.g., avoiding repeated summaries, questions, or informal language) are crucial and may not be fully captured by automatic metrics alone.

Key Results

  • The rule-based strapline detection method achieved good performance on a manually annotated test set. For Time.com, the model achieved a precision of 68% and a recall of 64%.
  • The strapline detector identified that 20.07% of NYT samples and 37.61% of Time.com samples were classified as straplines.
  • When straplines and noise were removed from the training data, summarization performance (measured by ROUGE scores) improved. For NYT, improvements reached up to 7 points in ROUGE score.
  • A case study showed that summaries generated from cleaned data were more informative and exhibited fewer undesirable strapline characteristics.

Limitations and Future Work

  • Generalizability: The authors acknowledge that their heuristics, developed on two specific news sources (NYT and Time.com), might not perform as well on other news sources with different styles or on datasets collected differently.
  • Future Work: The paper suggests investigating the effectiveness of their heuristics on a wider range of news sources and datasets. Further work could also explore more sophisticated methods for strapline detection beyond rule-based approaches.

Summary updated

BibTeX
@inproceedings{keleg2022automatically,
  author = {Amr Keleg and Matthias Lindemann and Danyang Liu and Wanqiu Long and Bonnie L. Webber},
  title = {{Automatically Discarding Straplines to Improve Data Quality for Abstractive News Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure