On Generating Extended Summaries of Long Documents

Sajad Sotudeh, Arman Cohan, Nazli Goharian

Introduction

The field of automated summarization has historically focused on generating short, abstract-like snippets. While these are valuable for quick scanning, they often fail to capture the technical nuances and detailed findings of long-form documents, such as 20-page research papers or complex legal filings. For a researcher needing to understand specific experimental conditions or a practitioner looking for methodology details without reading the entire text, an "extended summary" is required.

The primary technical challenge in summarizing long documents is known as "lead bias." Standard machine learning models often learn that the most important information is concentrated at the beginning of a document. Consequently, they tend to over-extract from the introduction while ignoring critical information in later sections. This paper presents a structural approach to move beyond the "lead" and capture the hierarchical organization of a document—Introduction, Methods, Results, and Conclusion—to create a more balanced and comprehensive summary.

Methodology: BERTSUMEXTMULTI

The researchers introduce BERTSUMEXTMULTI, a "section-aware" extractive summarizer designed to handle the complexities of long-form text. The model is built upon the BERTSUM architecture, which utilizes a BERT encoder to create high-quality representations of individual sentences within the context of the entire document.

The BERTSUMEXTMULTI architecture employs a multi-task learning framework, utilizing a shared Transformer Encoder to feed both sentence selection and section prediction heads.

The core innovation of this approach is the use of Multi-Task Learning. Instead of training the model solely on the primary task of sentence selection, the authors introduce an auxiliary task:

  • Task A (Sentence Selection): A binary classification task where the model predicts whether a given sentence should be included in the final summary.
  • Task B (Section Prediction): A multi-class classification task where the model identifies which section (e.g., Introduction, Experiments, Conclusion) a sentence belongs to.

The logic behind this dual-task approach is that document structure serves as a roadmap for importance. By forcing the model to identify the section, the researchers "nudge" it to pay attention to the document's internal hierarchy. This helps the model realize that a "Methods" section might contain summary-worthy content even if it appears ten pages into the document, effectively counteracting the tendency to focus only on the lead.

Datasets and Evaluation

The study evaluated the model's performance across three primary datasets:

  • Longsumm: A specialized dataset from a 2020 shared task designed for generating long summaries of scientific papers.
  • arXiv-Long & PubMed-Long: Two large-scale datasets curated by the authors. These were created by filtering existing corpora to include only papers with "long" ground-truth summaries (abstracts containing at least 350 tokens), providing a robust benchmark for extended summarization.

Key Results and Insights

The proposed model demonstrated significant success, ranking first in the Longsumm 2020 shared task based on F-measure, scoring 30.07 and outperforming several competitive systems.

ROUGE-2 scores on the Longsumm and arXiv-Long datasets demonstrate that the multi-tasking approach consistently improves summary quality across various lengths compared to the baseline.

Quantitative analysis revealed that the multi-tasking approach provided the most substantial performance boosts on the longest summaries. This suggests that as the target summary length grows, structural awareness becomes increasingly critical for maintaining coverage and quality.

Furthermore, qualitative analysis using heatmaps of sentence selection confirmed that the model successfully mitigated lead bias. While the baseline model focused almost entirely on the first 10% of the paper, the multi-tasking model identified salient sentences scattered throughout the middle and end of the documents. In one specific case study, the baseline model extracted 47% of its sentences from the very beginning, whereas the multi-tasking model sampled from diverse sections, resulting in a 20% jump in F1 score for that document.

Conclusion and Future Directions

Despite its success, the model has limitations. While it is adept at identifying which sections are important, it sometimes struggles with "salience within a section"—picking the single best sentence from a correctly identified relevant section.

Future work could explore "Dynamic Task Prioritization," where the model learns to weight the two tasks differently during training rather than using a fixed parameter. Additionally, the insights gained from this extractive model could be applied to abstractive summarization, where the model rewrites and synthesizes information in its own words while guided by the document's structural roadmap.

Summary updated

BibTeX
@inproceedings{sotudeh2021generating,
  author = {Sajad Sotudeh and Arman Cohan and Nazli Goharian},
  title = {{On Generating Extended Summaries of Long Documents}},
  booktitle = {SDU @ AAAI},
  year = {2021},
}
Figure