TL;DR
The "Summaformers" system tackles the challenge of summarizing dense scientific papers by leveraging their inherent sectional structure (e.g., Abstract, Introduction, Results). By selectively weighting these sections—focusing on the abstract for layman-friendly summaries and a "budgeted" extractive approach for long-form summaries—the authors secured 1st and 3rd place rankings in the EMNLP 2020 Scholarly Document Processing shared tasks.
What Problem Does This Paper Address?
As the volume of scientific literature explodes, both experts and laypeople struggle to keep up. Experts need detailed, long-form summaries to quickly grasp a paper's nuances without reading 10+ pages, while laypeople need jargon-free, concise explanations of a paper's core essence.
The authors address two specific tasks:
- LaySumm: Generating a short (~150 word) summary in plain language.
- LongSumm: Generating a detailed (~700+ word) summary that provides specific insights into the paper's various ideas.
The core challenge is "sectional importance": a paper's conclusion might be vital for a layperson, while the "Related Work" or "Results" sections are more critical for a researcher seeking a deep dive.
Methods
The authors developed two distinct pipelines based on the target summary type:
- For LaySumm (Abstractive): The team analyzed ROUGE overlaps (a metric for summary quality) and discovered that the original paper's Abstract is the most significant contributor to a layman summary. They treated this as a "style transfer" and summarization task, fine-tuning pre-trained Transformer models—specifically BART-large-CNN and T5—to rewrite the technical abstract into a simpler form.
- For LongSumm (Extractive): They used a hierarchical approach. First, they employed SummaRuNNer (a neural extractive summarizer) to pull key sentences from individual sections. To decide how many sentences to take from each part, they built a Budget Module. This module assigns weights to sections based on their historical importance (e.g., the Introduction and Results are weighted more heavily than Acknowledgments).
Datasets
The study utilized datasets provided by the 'Workshop on Scholarly Document Processing @ EMNLP 2020':
- LaySumm Dataset: 572 research papers from diverse domains (epilepsy, archeology, etc.) with corresponding gold-standard layman summaries.
- LongSumm Dataset: A larger corpus containing 1,705 extractive summaries (based on video talks) and 531 abstractive summaries (based on blog posts) of NLP and Machine Learning papers.
- Blind Test Sets: 37 papers for LaySumm and 22 for LongSumm were used for the final leaderboard rankings.
Key Insights and Ideas
- The "Abstract" Paradox: For LaySumm, the authors found that providing more context (like adding the Introduction or Conclusion to the model input) actually performed worse than just using the Abstract alone. The Abstract contains the most "verbatim" overlap with layman summaries, making it the most efficient source.
- Sectional Awareness is Key: For long summaries, the Introduction is the most critical section, followed by Related Work and Results. By ignoring "global context" and focusing purely on summarizing one section at a time before concatenating them, the model avoided becoming overwhelmed by the document's length.
- Heuristics Matter: Simple post-processing—like removing citations in brackets (e.g., "[12]") and cleaning up non-English Unicode characters—provided a measurable boost to the final summary quality.
Key Results
The Summaformers system proved highly effective in the shared task competitions:
- LongSumm: Ranked 1st overall with a ROUGE-1 F1 score of 49.46.
- LaySumm: Ranked 3rd overall with a ROUGE-1 F1 score of 45.94.
- Baseline Comparison: Interestingly, the authors noted that a "Lead-150" baseline (simply taking the first 150 tokens of an abstract) is surprisingly hard to beat for layman summaries, highlighting how much critical information authors pack into the very beginning of their papers.
Limitations and Future Work
The authors identified a few areas for improvement:
- Section Conflation: The current system treats "Methods" and "Materials and Methods" as different headers. They plan to implement "conflation" to group similar section types together for more consistent analysis.
- Deeper Sectional Analysis: While the "Budget Module" works well, the authors want to explore more sophisticated ways to determine which specific parts of a section are "summary-worthy" beyond simple ROUGE overlap.
- Pre-training: They intend to explore pre-training their extractive models on even larger, more diverse scientific datasets beyond PubMed.
Summary updated