Introduction
In the rapidly expanding landscape of academic research, staying current with the latest literature is a significant challenge. Survey papers serve as essential guides, yet their manual creation is labor-intensive and they quickly become obsolete. While Multi-Document Summarization (MDS) has advanced, existing models typically handle only a few documents or produce unstructured outputs. To address this, we introduce BigSurvey, a large-scale dataset, and CAST, a novel summarization framework designed to generate structured, multi-section summaries from dozens of reference papers.
The BigSurvey Dataset
BigSurvey is the first large-scale dataset specifically curated for the automated generation of survey papers. It comprises over 7,000 survey papers from arXiv and metadata for 434,000 referenced works. The dataset is divided into two primary tasks to mimic the human writing process:
- BigSurvey-MDS: Focuses on generating structured introductions (Background, Method, Other) from an average of 76 reference abstracts per sample.
- BigSurvey-Abs: Focuses on generating a concise abstract from the full body text of a survey paper.
Compared to traditional MDS datasets like Multi-News, BigSurvey-MDS involves significantly longer input sequences and a higher volume of source documents, presenting a unique challenge for long-context modeling.
The CAST Framework
To process the massive input sequences (averaging 12,000 words) and organize diverse information, we propose CAST (Category-based Alignment and Sparse Transformer). The framework introduces two key innovations:
Category-Based Alignment (CA)
Standard models often struggle with noise when processing dozens of abstracts. CAST utilizes a BERT-based classifier to label input sentences as Background, Method, or Other. By aligning these categories with the target sections of the summary, the model effectively filters irrelevant information, ensuring that the "Method" section of the summary is primarily informed by "Method" sentences from the source text.
Sparse Transformers
To overcome the quadratic scaling limitations of standard Transformers, CAST employs Sparse Transformers (such as Longformer-Encoder-Decoder or BigBird). These architectures scale linearly with sequence length, allowing the model to efficiently process the extensive context required for comprehensive literature synthesis.
Results and Key Insights
Our evaluations demonstrate that CAST-LED (Longformer-Encoder-Decoder) achieves state-of-the-art performance on the BigSurvey-MDS dataset, significantly outperforming baselines like BART and PEGASUS. Key findings include:
- The Necessity of Alignment: Ablation studies revealed that removing Category-Based Alignment leads to a substantial drop in ROUGE scores (ROUGE-1 falling from 43.13 to 39.79). This confirms that simply increasing context windows is insufficient; models require structural guidance to manage long-form content.
- Abstracts as Proxies: The success of the model using only reference abstracts suggests that these summaries contain enough dense information to reconstruct a field's narrative.
- Human Preference: In blind evaluations, CAST-LED was consistently rated higher for informativeness and non-redundancy compared to standard long-context models.
Conclusion
CAST and the BigSurvey dataset represent a significant step toward automated literature review generation. While current models excel at extraction and organization, they still lack the critical reasoning and evaluative depth of human experts. Future research will focus on incorporating scientific critique and deeper reasoning capabilities into the summarization process.
Summary updated