Summaformers @ LaySumm 20, LongSumm 20

Sayar Ghosh Roy, Nikhil Pinnaparaju, Risubh Jain, Manish Gupta, Vasudeva Varma

Abstract

Automatic text summarization has been widely studied as an important task in natural language processing. Traditionally, various feature engineering and machine learning based systems have been proposed for extractive as well as abstractive text summarization. Recently, deep learning based, specifically Transformer-based systems have been immensely popular. Summarization is a cognitively challenging task - extracting summary worthy sentences is laborious, and expressing semantics in brief when doing abstractive summarization is complicated. In this paper, we specifically look at the problem of summarizing scientific research papers from multiple domains. We differentiate between two types of summaries, namely, (a) LaySumm: A very short summary that captures the essence of the research paper in layman terms restricting overtly specific technical jargon and (b) LongSumm: A much longer detailed summary aimed at providing specific insights into various ideas touched upon in the paper. While leveraging latest Transformer-based models, our systems are simple, intuitive and based on how specific paper sections contribute to human summaries of the two types described above. Evaluations against gold standard summaries using ROUGE metrics prove the effectiveness of our approach. On blind test corpora, our system ranks first and third for the LongSumm and LaySumm tasks respectively.

Introduction

As the volume of scientific literature continues to expand exponentially, the need for efficient summarization tools has become critical for both experts and the general public. Experts require detailed, long-form summaries to navigate the nuances of a paper without committing to a full 10+ page reading, while laypeople need jargon-free, concise explanations of a paper's core contributions. The "Summaformers" system addresses these needs by participating in two specific tasks from the EMNLP 2020 Scholarly Document Processing workshop: LaySumm (generating a ~150-word layman summary) and LongSumm (generating a ~700-word detailed summary).

The core hypothesis of this work is "sectional importance." Different sections of a scientific paper—such as the Abstract, Introduction, or Results—contribute differently to the final summary depending on the target audience and length requirements.

Methodology

The authors developed two distinct pipelines tailored to the specific requirements of each task.

LaySumm (Abstractive Pipeline)

For layman summaries, analysis revealed that the original paper's Abstract is the most significant contributor. The team treated this as a combination of "style transfer" and summarization. They fine-tuned pre-trained Transformer models, specifically BART-large-CNN and T5, to rewrite technical abstracts into simplified, layman-friendly language.

LongSumm (Extractive Pipeline)

For long-form summaries, a hierarchical approach was employed. The system first uses SummaRuNNer, a neural extractive summarizer, to identify key sentences within individual sections. These sectional summaries are then aggregated through a Budget Module. This module assigns weights to sections based on their historical importance; for instance, the Introduction and Results are prioritized over Acknowledgments or Financial Support.

The Summaformers hierarchical pipeline for LongSumm, utilizing a Budget Module to aggregate sectional summaries.

Sectional Importance and Insights

A key finding of the study is the "Abstract Paradox" in LaySumm. The authors discovered that providing the model with more context—such as adding the Introduction or Conclusion to the input—actually degraded performance compared to using the Abstract alone. As shown in the match score analysis, the Abstract contains the highest verbatim overlap with layman summaries, making it the most efficient source for simplification.

Match scores across paper sections for LaySumm, highlighting the dominance of the abstract.

Conversely, for LongSumm, the Introduction was found to be the most critical section, followed by Related Work and Results. By processing these sections independently before concatenation, the model avoided the computational and contextual overwhelm often associated with processing extremely long documents.

Results and Performance

The Summaformers system achieved top-tier results in the shared task competitions:

  • LongSumm: Ranked 1st overall with a ROUGE-1 F1 score of 49.46.
  • LaySumm: Ranked 3rd overall with a ROUGE-1 F1 score of 45.94.

Interestingly, the authors noted that a simple "Lead-150" baseline—taking the first 150 tokens of an abstract—is remarkably difficult to beat for layman summaries. This underscores how effectively researchers pack critical information into the opening lines of their abstracts. Additionally, simple post-processing heuristics, such as removing bracketed citations and cleaning Unicode characters, provided measurable boosts to the final ROUGE scores.

Conclusion and Future Work

While the current system is highly effective, the authors identified several paths for future improvement. They plan to implement "section conflation" to group similar headers (e.g., "Methods" and "Materials and Methods") for more consistent analysis. Furthermore, they aim to refine the Budget Module to determine summary-worthiness beyond simple ROUGE overlaps and explore pre-training extractive models on larger, more diverse scientific datasets like PubMed.

Summary updated

Abstract

Automatic text summarization has been widely studied as an important task in natural language processing. Traditionally, various feature engineering and machine learning based systems have been proposed for extractive as well as abstractive text summarization. Recently, deep learning based, specifically Transformer-based systems have been immensely popular. Summarization is a cognitively challenging task - extracting summary worthy sentences is laborious, and expressing semantics in brief when doing abstractive summarization is complicated. In this paper, we specifically look at the problem of summarizing scientific research papers from multiple domains. We differentiate between two types of summaries, namely, (a) LaySumm: A very short summary that captures the essence of the research paper in layman terms restricting overtly specific technical jargon and (b) LongSumm: A much longer detailed summary aimed at providing specific insights into various ideas touched upon in the paper. While leveraging latest Transformer-based models, our systems are simple, intuitive and based on how specific paper sections contribute to human summaries of the two types described above. Evaluations against gold standard summaries using ROUGE metrics prove the effectiveness of our approach. On blind test corpora, our system ranks first and third for the LongSumm and LaySumm tasks respectively.
BibTeX
@inproceedings{roy2020summaformers,
  author = {Sayar Ghosh Roy and Nikhil Pinnaparaju and Risubh Jain and Manish Gupta and Vasudeva Varma},
  title = {{Summaformers @ LaySumm 20, LongSumm 20}},
  booktitle = {SDP @ EMNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=M6CfTffDaE},
}
Figure