Title-based Extractive Summarization via MRC Framework

Hongjin Kim, Jai-Eun Kim, Harksoo KimCOLING 2024

TL;DR

This paper introduces MRCSUM, a novel approach to extractive text summarization that frames the task as a Machine Reading Comprehension (MRC) problem. By treating the document's title as a query, MRCSUM effectively captures the document's core semantics, leading to more coherent and informative summaries, even when titles are unavailable.

What Problem Does This Paper Address?

Traditional extractive summarization methods often focus on scoring and selecting sentences independently. This can lead to summaries that are overly generalized and fail to capture the overall essence of a document. While some approaches have tried to incorporate document-level semantics, they often struggle with situations where a clear title or explicit query is missing. The authors aim to bridge this gap by leveraging the title as a semantic anchor for summarization, proposing a robust framework that can also handle title-less documents.

Methods

The core innovation of MRCSUM lies in its formulation of extractive summarization as a Machine Reading Comprehension (MRC) task. Instead of traditional sentence scoring, MRCSUM uses the document's title as a "TITLE QUERY" to guide an MRC model. This model then extracts "candidate summary spans" that are semantically relevant to the title.

When a title is not available, MRCSUM generates a "TITLE-LIKE QUERY" by combining topic modeling (using LDA) and keyword extraction (using KeyBERT and TextRank). This TITLE-LIKE QUERY aims to mimic the semantic function of a title.

The model architecture builds upon pre-trained language models (PLMs) like BERT and RoBERTa. It encodes both the query (title or TITLE-LIKE QUERY) and the document. A key component is the "span matrix," inspired by recent MRC advancements, which allows for the effective extraction of multiple summary sentences, addressing the limitation of single-sentence extraction in prior work. The training involves minimizing losses related to start and end index prediction for summary sentences, as well as a span pairing loss.

Datasets

The authors evaluated MRCSUM on several standard datasets:

  • English: CNN/DailyMail, Reddit, XSum.
  • Korean: Modu-news (a newly constructed dataset from the Modu-corpus).

They followed existing data-labeling practices for these datasets and specifically noted that Modu-news includes document titles, which is crucial for their TITLE QUERY approach.

Key Insights and Ideas

  • Title as a Semantic Anchor: The paper highlights that a document's title, when used as a query, is a powerful semantic guide for extractive summarization, leading to more coherent and relevant summaries than sentence-level scoring alone.
  • TITLE-LIKE QUERY for Title-less Documents: The ability to generate a meaningful query from topics and keywords when a title is absent is a significant contribution, making the approach more broadly applicable.
  • MRC Framework for Summarization: Framing summarization as an MRC task, specifically by extracting answer spans based on a title query, offers a novel perspective that leverages the strengths of modern NLP models.
  • Span Matrix for Multi-Sentence Extraction: The adoption of a span matrix is crucial for effectively extracting multiple sentences, a common requirement for comprehensive summaries, and overcomes limitations of prior models.
  • TextRank for Keyword Extraction: The study found that TextRank outperformed KeyBERT for keyword extraction in their context, likely due to its ability to handle longer documents and avoid token limitations.

Key Results

  • MRCSUM significantly outperformed several state-of-the-art baselines (e.g., MATCHSUM, SIDENET, BERTEXT) on both Korean (Modu-news) and English (CNN/DM, Reddit, XSum) datasets, achieving higher ROUGE scores.
  • On the Modu-news dataset, MRCSUM (using TITLE QUERY) achieved an ROUGE-L score of 44.03, outperforming other models.
  • MRCSUM with TITLE-LIKE QUERY also showed strong performance, demonstrating the effectiveness of the generated query.
  • Human evaluation on Modu-news and CNN/DM datasets showed that MRCSUM summaries were rated as significantly more informative than those from baseline models, with scores comparable to the ground truth (ORACLE).
  • Ablation studies confirmed the importance of both the query generation (TITLE QUERY/TITLE-LIKE QUERY) and the span matrix for achieving high performance.

Limitations and Future Work

  • Title Enticement vs. Summary: The authors acknowledge that titles can sometimes be used for enticement rather than as a direct summary. While their TITLE-LIKE QUERY can still be practical in such cases, it might not be as effective as a true title.
  • Comparison with LLMs: The paper notes that they did not directly compare MRCSUM with large language models (LLMs), which are currently trending. They point out that LLMs often rely on generation rather than extraction and have significantly larger model sizes and training data requirements, making direct resource-comparisons challenging.
  • Future Work: The authors plan to explore distinguishing between news titles that are summaries versus those that are teasers. They also aim to expand their work to effective summary extraction for both query-based and general summarization tasks.

Summary updated

BibTeX
@inproceedings{kim2024title,
  author = {Hongjin Kim and Jai-Eun Kim and Harksoo Kim},
  title = {{Title-based Extractive Summarization via MRC Framework}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure