Disentangling Instructive Information from Ranked Multiple Candidates for Multi-Document Scientific Summarization

Pancheng Wang, Shasha Li, Dong Li, Kehan Long, Jintao Tang, Ting Wang

TL;DR

The DIR (Disentangling Instructive information from Ranked candidates) framework improves multi-document scientific summarization by using summaries from other models as "external guides." By ranking these candidates and disentangling their content into positive signals (what to include) and negative signals (what to avoid) using a Conditional Variational Autoencoder, DIR provides the decoder with the global context and fine-grained control needed to produce state-of-the-art summaries.

What Problem Does This Paper Address?

Multi-Document Scientific Summarization (MDSS) is a high-stakes task: it requires condensing multiple, often dense, research papers into a single, coherent summary. Current abstractive methods, typically based on the Transformer architecture, face two major hurdles:

  1. Global Context Blindness: Decoders struggle to maintain a "big picture" view when processing diverse information scattered across multiple long documents.
  2. Lack of Decoding Guidance: While decoders are good at being fluent, they often lack a "plan." Without external guidance, they struggle to decide which specific discourse structures or content pieces to prioritize, leading to summaries that might miss the mark or include irrelevant details.

The authors argue that instead of just looking at the source documents, models should look at other summaries (candidates) to learn what a good summary looks like—and just as importantly, what a bad one looks like.

Methods

The authors propose the DIR framework, which can be plugged into various Transformer-based models. It consists of three main modules:

  1. Candidates Ranking: The system first generates multiple summary candidates using different models (like BART or LLMs). A specialized pairwise ranking module then identifies the highest-quality candidates. The authors found that selecting the "Top-K" candidates is more effective than simply using all of them.
  2. Instructive Information Modeling and Disentangling: This is the core innovation. Using a Conditional Variational Autoencoder (CVAE), the framework maps candidates into two distinct latent variables:
  3. Positive Information ($z_p$): Content that overlaps with the ground-truth "gold" summary.
  4. Negative Information ($z_n$): Content present in the candidate but absent from the gold summary. To ensure these signals don't mix, they use an informativeness objective (to make the variables predictive) and an adversarial objective (to ensure $z_p$ doesn't contain $z_n$ info, and vice versa).

  5. Information Augmented Decoding: The decoder is modified in two ways. First, it concatenates the candidate representations with the source document representations. Second, it "injects" the squeezed positive and negative latent variables directly into the cross-attention layer, effectively telling the decoder: "Focus more on these concepts, and suppress these others."

Figure 3: The overall framework of our proposed Disentangling Instructive information from Ranked candidates (DIR).

Datasets

The researchers evaluated DIR on three large-scale MDSS benchmarks:

  • Multi-Xscience: A dataset of over 40,000 instances where the goal is to summarize a query paper's abstract and its cited references into a "Related Work" paragraph.
  • TAD: A multi-field dataset collected from S2ORC.
  • TAS2: A computer science-focused dataset collected from the Delve corpus.

Key Insights and Ideas

  • Candidates as Dual Signals: A summary candidate isn't just "good" or "bad." It's a mixture of helpful matches and misleading mismatches. By treating mismatches as negative signals, the model learns what to avoid, which is just as important for quality as knowing what to include. Figure 1 illustrates how a candidate can have both matched (positive) and mismatched (negative) content compared to a gold summary.

  • The Power of Ranking: Simply concatenating all available candidates can actually hurt performance because it dilutes the role of the source papers. Selecting the top 3 candidates ($K=3$) was found to be the "sweet spot" for performance.

  • LLMs as Cheap Teachers: While training models like BART to generate candidates is effective, using a Large Language Model (like GPT-3.5) to generate candidates via prompting is a highly efficient way to get high-quality external guidance without heavy retraining.
Figure 1: The summary candidate may match the gold sum- mary in some aspects and not in others.

Key Results

The DIR framework consistently outperformed base models across all datasets and metrics (ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore).

  • New SOTA: The combination of KGSum + DIR achieved a new state-of-the-art performance on all three datasets, surpassing the previous leader, EDITSum.
  • Human Preference: In blind tests, human evaluators significantly preferred summaries generated by DIR-equipped models. For the TransS2S model, the DIR version was preferred 59.1% of the time, compared to only 21.1% for the base model. Human evaluation results showing a clear preference for DIR-equipped models across different architectures.

  • Disentanglement Success: Visualization of the latent space (using t-SNE) confirmed that the positive and negative information were successfully separated into distinct clusters after training.

Human evaluation results showing a clear preference for DIR-equipped models across different architectures.

Limitations and Future Work

  • Computational Overhead: Generating multiple candidates and ranking them adds a second stage to the inference process, which increases computational cost and latency.
  • Candidate Quality Dependency: The framework's success relies on the quality of the first-stage models. If all candidates are extremely poor, the "positive" signal may be too weak to be useful.
  • Future Directions: The authors suggest exploring more diverse ways to generate candidates and investigating how this "positive/negative" disentanglement might apply to other conditional text generation tasks beyond scientific summarization.

Summary updated

Abstract

Automatically condensing multiple topic-related scientific papers into a succinct and concise summary is referred to as Multi-Document Scientific Summarization (MDSS). Currently, while commonly used abstractive MDSS methods can generate flexible and coherent summaries, the difficulty in handling global information and the lack of guidance during decoding still make it challenging to generate better summaries. To alleviate these two shortcomings, this paper introduces summary candidates into MDSS, utilizing the global information of the document set and additional guidance from the summary candidates to guide the decoding process. Our insights are twofold: Firstly, summary candidates can provide instructive information from both positive and negative perspectives, and secondly, selecting higher-quality candidates from multiple options contributes to producing better summaries. Drawing on the insights, we propose a summary candidates fusion framework -- Disentangling Instructive information from Ranked candidates (DIR) for MDSS. Specifically, DIR first uses a specialized pairwise comparison method towards multiple candidates to pick out those of higher quality. Then DIR disentangles the instructive information of summary candidates into positive and negative latent variables with Conditional Variational Autoencoder. These variables are further incorporated into the decoder to guide generation. We evaluate our approach with three different types of Transformer-based models and three different types of candidates, and consistently observe noticeable performance improvements according to automatic and human evaluation. More analyses further demonstrate the effectiveness of our model in handling global information and enhancing decoding controllability.
BibTeX
@inproceedings{wang2024disentangling,
  author = {Pancheng Wang and Shasha Li and Dong Li and Kehan Long and Jintao Tang and Ting Wang},
  title = {{Disentangling Instructive Information from Ranked Multiple Candidates for Multi-Document Scientific Summarization}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
}
Figure