Disentangling Instructive Information from Ranked Multiple Candidates for Multi-Document Scientific Summarization

Pancheng Wang, Shasha Li, Dong Li, Kehan Long, Jintao Tang, Ting Wang

Abstract

Multi-Document Scientific Summarization (MDSS) requires synthesizing information from multiple diverse sources into a coherent summary. This task is hindered by challenges in maintaining global context and controlling content structure. We present DIR (Disentangling Instructive information from Ranked candidates), a framework that utilizes multiple pre-generated candidate summaries as external guidance. DIR ranks these candidates and employs a Conditional Variational Autoencoder (CVAE) to disentangle them into positive and negative latent signals. These signals are then injected into the decoding process to steer generation. Our approach achieves new state-of-the-art results on Multi-Xscience, TAD, and TAS2 datasets, with significant improvements in both automated metrics and human preference.

Introduction

Summarizing a collection of related scientific papers is significantly more complex than single-document summarization. Current abstractive models face two primary hurdles: maintaining a "big picture" view across thousands of tokens and effectively planning content to avoid redundancy. While internal features like knowledge graphs have been explored, this research argues that external guidance from pre-generated "summary candidates" provides a more direct way to steer the final output.

The DIR Framework

The DIR framework acts as a second-stage refiner for existing summarization models. It consists of three integrated modules designed to filter and utilize candidate information effectively.

1. Candidates Ranking Module

The framework first generates multiple candidate summaries using base models (e.g., BART or GPT-3.5). A specialized pairwise comparison method ranks these candidates, ensuring that the subsequent modules prioritize high-quality guidance and ignore noisy or low-quality inputs.

2. Instructive Information Modeling and Disentangling

At the heart of DIR is the disentanglement of candidate information. Using a Conditional Variational Autoencoder (CVAE), the model identifies "Positive Information" (content overlapping with the ground truth) and "Negative Information" (irrelevant or redundant content). Adversarial training ensures these signals are mathematically separated into distinct latent variables.

3. Information Augmented Decoding

The final decoder incorporates these signals in two ways:

  • Contextual Concatenation: The candidate text is processed alongside the source documents.
  • Latent Injection: The disentangled variables are injected into the cross-attention layer, effectively "adding" positive signals and "subtracting" negative ones to guide word generation.
The DIR framework architecture, comprising candidate ranking, information disentangling, and augmented decoding.

Experimental Results

The DIR framework was evaluated on three major MDSS benchmarks: Multi-Xscience (arXiv/MAG related work), TAD (multi-field S2ORC), and TAS2 (computer science Delve).

Quantitative Performance

DIR consistently improved the performance of various base models, including TransS2S, KGSum, and BART. Notably, KGSum + DIR established a new state-of-the-art on all three datasets, outperforming previous leaders like EDITSum. On the Multi-Xscience dataset, DIR provided an average ROUGE score improvement of 7.42% over the base TransS2S model.

Human Evaluation

Blind human evaluations confirmed the superiority of DIR-enhanced summaries. Evaluators consistently preferred summaries generated with DIR over base models, citing better organization and reduced redundancy. For instance, human evaluators showed a 59.1% preference for TransS2S+DIR compared to only 21.1% for the base version.

Human evaluation results comparing DIR-equipped models against their base versions across three datasets.

Discussion and Conclusion

A key insight of this work is "The Power of 'No'": explicitly modeling negative information is as crucial as identifying positive content for preventing hallucinations in scientific text. Furthermore, the study demonstrates that Large Language Models (LLMs), while not always the best direct summarizers for specific scientific tasks, serve as excellent "teachers" by providing high-quality candidates for refinement.

Despite the added computational overhead during inference, the DIR framework offers a robust path forward for controllable and high-quality multi-document summarization. Future work will explore applying this disentanglement approach to larger models and broader stylistic controls.

Summary updated

Abstract

Automatically condensing multiple topic-related scientific papers into a succinct and concise summary is referred to as Multi-Document Scientific Summarization (MDSS). Currently, while commonly used abstractive MDSS methods can generate flexible and coherent summaries, the difficulty in handling global information and the lack of guidance during decoding still make it challenging to generate better summaries. To alleviate these two shortcomings, this paper introduces summary candidates into MDSS, utilizing the global information of the document set and additional guidance from the summary candidates to guide the decoding process. Our insights are twofold: Firstly, summary candidates can provide instructive information from both positive and negative perspectives, and secondly, selecting higher-quality candidates from multiple options contributes to producing better summaries. Drawing on the insights, we propose a summary candidates fusion framework -- Disentangling Instructive information from Ranked candidates (DIR) for MDSS. Specifically, DIR first uses a specialized pairwise comparison method towards multiple candidates to pick out those of higher quality. Then DIR disentangles the instructive information of summary candidates into positive and negative latent variables with Conditional Variational Autoencoder. These variables are further incorporated into the decoder to guide generation. We evaluate our approach with three different types of Transformer-based models and three different types of candidates, and consistently observe noticeable performance improvements according to automatic and human evaluation. More analyses further demonstrate the effectiveness of our model in handling global information and enhancing decoding controllability.
BibTeX
@inproceedings{wang2024disentangling,
  author = {Pancheng Wang and Shasha Li and Dong Li and Kehan Long and Jintao Tang and Ting Wang},
  title = {{Disentangling Instructive Information from Ranked Multiple Candidates for Multi-Document Scientific Summarization}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
}
Figure