Generating a Structured Summary of Numerous Academic Papers: Dataset and Method

Shuaiqi Liu, Jiannong Cao, Ruosong Yang, Zhiyuan Wen

Introduction

In the rapidly expanding landscape of academic research, staying current with the latest literature is a significant challenge. Survey papers serve as essential guides, yet their manual creation is labor-intensive and they quickly become obsolete. While Multi-Document Summarization (MDS) has advanced, existing models typically handle only a few documents or produce unstructured outputs. To address this, we introduce BigSurvey, a large-scale dataset, and CAST, a novel summarization framework designed to generate structured, multi-section summaries from dozens of reference papers.

The BigSurvey Dataset

BigSurvey is the first large-scale dataset specifically curated for the automated generation of survey papers. It comprises over 7,000 survey papers from arXiv and metadata for 434,000 referenced works. The dataset is divided into two primary tasks to mimic the human writing process:

  1. BigSurvey-MDS: Focuses on generating structured introductions (Background, Method, Other) from an average of 76 reference abstracts per sample.
  2. BigSurvey-Abs: Focuses on generating a concise abstract from the full body text of a survey paper.
Extractive characteristics of BigSurvey subsets.

Compared to traditional MDS datasets like Multi-News, BigSurvey-MDS involves significantly longer input sequences and a higher volume of source documents, presenting a unique challenge for long-context modeling.

The CAST Framework

To process the massive input sequences (averaging 12,000 words) and organize diverse information, we propose CAST (Category-based Alignment and Sparse Transformer). The framework introduces two key innovations:

Category-Based Alignment (CA)

Standard models often struggle with noise when processing dozens of abstracts. CAST utilizes a BERT-based classifier to label input sentences as Background, Method, or Other. By aligning these categories with the target sections of the summary, the model effectively filters irrelevant information, ensuring that the "Method" section of the summary is primarily informed by "Method" sentences from the source text.

Sparse Transformers

To overcome the quadratic scaling limitations of standard Transformers, CAST employs Sparse Transformers (such as Longformer-Encoder-Decoder or BigBird). These architectures scale linearly with sequence length, allowing the model to efficiently process the extensive context required for comprehensive literature synthesis.

The CAST architecture pipeline.

Results and Key Insights

Our evaluations demonstrate that CAST-LED (Longformer-Encoder-Decoder) achieves state-of-the-art performance on the BigSurvey-MDS dataset, significantly outperforming baselines like BART and PEGASUS. Key findings include:

  • The Necessity of Alignment: Ablation studies revealed that removing Category-Based Alignment leads to a substantial drop in ROUGE scores (ROUGE-1 falling from 43.13 to 39.79). This confirms that simply increasing context windows is insufficient; models require structural guidance to manage long-form content.
  • Abstracts as Proxies: The success of the model using only reference abstracts suggests that these summaries contain enough dense information to reconstruct a field's narrative.
  • Human Preference: In blind evaluations, CAST-LED was consistently rated higher for informativeness and non-redundancy compared to standard long-context models.

Conclusion

CAST and the BigSurvey dataset represent a significant step toward automated literature review generation. While current models excel at extraction and organization, they still lack the critical reasoning and evaluative depth of human experts. Future research will focus on incorporating scientific critique and deeper reasoning capabilities into the summarization process.

Summary updated

Abstract

Writing a survey paper on one research topic usually needs to cover the salient content from numerous related papers, which can be modeled as a multi-document summarization (MDS) task. Existing MDS datasets usually focus on producing the structureless summary covering a few input documents. Meanwhile, previous structured summary generation works focus on summarizing a single document into a multi-section summary. These existing datasets and methods cannot meet the requirements of summarizing numerous academic papers into a structured summary. To deal with the scarcity of available data, we propose BigSurvey, the first large-scale dataset for generating comprehensive summaries of numerous academic papers on each topic. We collect target summaries from more than seven thousand survey papers and utilize their 430 thousand reference papers' abstracts as input documents. To organize the diverse content from dozens of input documents and ensure the efficiency of processing long text sequences, we propose a summarization method named category-based alignment and sparse transformer (CAST). The experimental results show that our CAST method outperforms various advanced summarization methods.
BibTeX
@inproceedings{liu2022generating,
  author = {Shuaiqi Liu and Jiannong Cao and Ruosong Yang and Zhiyuan Wen},
  title = {{Generating a Structured Summary of Numerous Academic Papers: Dataset and Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/591},
}
Figure