Generating a Structured Summary of Numerous Academic Papers: Dataset and Method

Shuaiqi Liu, Jiannong Cao, Ruosong Yang, Zhiyuan Wen

Introduction

Writing a survey paper is a fundamental yet labor-intensive task for researchers. While Multi-Document Summarization (MDS) has seen significant progress, applying it to the academic domain reveals three critical gaps: data scarcity (most datasets use fewer than 10 documents), lack of structure (outputs are often monolithic blocks), and the "long context" problem (processing dozens of papers exceeds standard Transformer memory limits). To address these challenges, researchers from The Hong Kong Polytechnic University introduced BigSurvey, a large-scale dataset, and CAST, a novel summarization framework designed for high-volume academic inputs.

The BigSurvey Dataset

BigSurvey is designed to mirror the actual complexity of survey writing, moving beyond the small-scale inputs of previous datasets like Multi-XScience. It consists of two primary subsets:

  • BigSurvey-MDS: Over 4,400 examples requiring the generation of structured summaries (Background, Method, Other) from an average of 76.3 reference abstracts.
  • BigSurvey-Abs: 7,123 examples targeting concise survey-level abstracts.
Extractive fragment density and coverage for the MDS-Other and ABS subsets of BigSurvey.

As shown in the dataset analysis, these summaries maintain high extractive coverage while managing varying levels of density, providing a realistic testbed for large-scale academic summarization.

The CAST Framework

The proposed CAST (Category-based Alignment and Sparse Transformer) method tackles the noise inherent in long-context inputs through a two-step "denoising" process involving content selection and efficient attention mechanisms.

Category-based Alignment (CA)

Instead of a naive "one-to-many" approach where all source text is fed into a single generator, CAST uses sentence classifiers (SSC and SCC) to categorize input sentences by function (e.g., Background, Method, Objective). These are then aligned with the corresponding target sections of the survey. This content selection ensures the model focuses on relevant source material for each specific section it generates, preventing "content mixing" where sections overlap or repeat information.

Sparse Transformers

To handle inputs averaging 12,000 words, CAST utilizes sparse attention mechanisms found in models like BigBird and Longformer (LED). These architectures scale linearly with sequence length, preventing the quadratic memory explosion typical of standard Transformers and allowing the model to process dozens of abstracts simultaneously.

The CAST framework architecture, showing the pipeline from reference papers to structured summaries via sentence classification and sparse transformers.

Key Results and Insights

The CAST-LED variant achieved a Combined R-1 score of 43.13 on the BigSurvey-MDS test set, significantly outperforming standard BART (37.64) and PEGASUS (38.91) baselines. Ablation studies highlighted the importance of the alignment step: removing Category-based Alignment caused the R-1 score to drop to 39.79, confirming that effective content selection is as vital as the underlying model architecture.

Human evaluations further supported these findings, with annotators rating CAST-generated summaries higher for informativeness and lower for redundancy compared to standard long-context models. The "aha moment" of the research is that for very long MDS, the primary challenge is content selection rather than just generation.

Conclusion and Future Directions

While CAST excels at aggregating and organizing information, current models still struggle with the "critical reasoning" required for high-level synthesis. A human-written survey critiques and synthesizes, whereas current models remain largely extractive or aggregative. Future research will likely expand from abstracts to full-text papers, further pushing the boundaries of long-context alignment and sparse attention to capture the full depth of academic discourse.

Summary updated

BibTeX
@inproceedings{liu2022generating,
  author = {Shuaiqi Liu and Jiannong Cao and Ruosong Yang and Zhiyuan Wen},
  title = {{Generating a Structured Summary of Numerous Academic Papers: Dataset and Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/591},
}
Figure