Abstract
Multi-Document Scientific Summarization (MDSS) requires synthesizing information from multiple diverse sources into a coherent summary. This task is hindered by challenges in maintaining global context and controlling content structure. We present DIR (Disentangling Instructive information from Ranked candidates), a framework that utilizes multiple pre-generated candidate summaries as external guidance. DIR ranks these candidates and employs a Conditional Variational Autoencoder (CVAE) to disentangle them into positive and negative latent signals. These signals are then injected into the decoding process to steer generation. Our approach achieves new state-of-the-art results on Multi-Xscience, TAD, and TAS2 datasets, with significant improvements in both automated metrics and human preference.
Introduction
Summarizing a collection of related scientific papers is significantly more complex than single-document summarization. Current abstractive models face two primary hurdles: maintaining a "big picture" view across thousands of tokens and effectively planning content to avoid redundancy. While internal features like knowledge graphs have been explored, this research argues that external guidance from pre-generated "summary candidates" provides a more direct way to steer the final output.
The DIR Framework
The DIR framework acts as a second-stage refiner for existing summarization models. It consists of three integrated modules designed to filter and utilize candidate information effectively.
1. Candidates Ranking Module
The framework first generates multiple candidate summaries using base models (e.g., BART or GPT-3.5). A specialized pairwise comparison method ranks these candidates, ensuring that the subsequent modules prioritize high-quality guidance and ignore noisy or low-quality inputs.
2. Instructive Information Modeling and Disentangling
At the heart of DIR is the disentanglement of candidate information. Using a Conditional Variational Autoencoder (CVAE), the model identifies "Positive Information" (content overlapping with the ground truth) and "Negative Information" (irrelevant or redundant content). Adversarial training ensures these signals are mathematically separated into distinct latent variables.
3. Information Augmented Decoding
The final decoder incorporates these signals in two ways:
- Contextual Concatenation: The candidate text is processed alongside the source documents.
- Latent Injection: The disentangled variables are injected into the cross-attention layer, effectively "adding" positive signals and "subtracting" negative ones to guide word generation.
Experimental Results
The DIR framework was evaluated on three major MDSS benchmarks: Multi-Xscience (arXiv/MAG related work), TAD (multi-field S2ORC), and TAS2 (computer science Delve).
Quantitative Performance
DIR consistently improved the performance of various base models, including TransS2S, KGSum, and BART. Notably, KGSum + DIR established a new state-of-the-art on all three datasets, outperforming previous leaders like EDITSum. On the Multi-Xscience dataset, DIR provided an average ROUGE score improvement of 7.42% over the base TransS2S model.
Human Evaluation
Blind human evaluations confirmed the superiority of DIR-enhanced summaries. Evaluators consistently preferred summaries generated with DIR over base models, citing better organization and reduced redundancy. For instance, human evaluators showed a 59.1% preference for TransS2S+DIR compared to only 21.1% for the base version.
Discussion and Conclusion
A key insight of this work is "The Power of 'No'": explicitly modeling negative information is as crucial as identifying positive content for preventing hallucinations in scientific text. Furthermore, the study demonstrates that Large Language Models (LLMs), while not always the best direct summarizers for specific scientific tasks, serve as excellent "teachers" by providing high-quality candidates for refinement.
Despite the added computational overhead during inference, the DIR framework offers a robust path forward for controllable and high-quality multi-document summarization. Future work will explore applying this disentanglement approach to larger models and broader stylistic controls.
Summary updated