TL;DR
The DIR (Disentangling Instructive information from Ranked candidates) framework improves multi-document scientific summarization by using summaries from other models as "external guides." By ranking these candidates and disentangling their content into positive signals (what to include) and negative signals (what to avoid) using a Conditional Variational Autoencoder, DIR provides the decoder with the global context and fine-grained control needed to produce state-of-the-art summaries.
What Problem Does This Paper Address?
Multi-Document Scientific Summarization (MDSS) is a high-stakes task: it requires condensing multiple, often dense, research papers into a single, coherent summary. Current abstractive methods, typically based on the Transformer architecture, face two major hurdles:
- Global Context Blindness: Decoders struggle to maintain a "big picture" view when processing diverse information scattered across multiple long documents.
- Lack of Decoding Guidance: While decoders are good at being fluent, they often lack a "plan." Without external guidance, they struggle to decide which specific discourse structures or content pieces to prioritize, leading to summaries that might miss the mark or include irrelevant details.
The authors argue that instead of just looking at the source documents, models should look at other summaries (candidates) to learn what a good summary looks like—and just as importantly, what a bad one looks like.
Methods
The authors propose the DIR framework, which can be plugged into various Transformer-based models. It consists of three main modules:
- Candidates Ranking: The system first generates multiple summary candidates using different models (like BART or LLMs). A specialized pairwise ranking module then identifies the highest-quality candidates. The authors found that selecting the "Top-K" candidates is more effective than simply using all of them.
- Instructive Information Modeling and Disentangling: This is the core innovation. Using a Conditional Variational Autoencoder (CVAE), the framework maps candidates into two distinct latent variables:
- Positive Information ($z_p$): Content that overlaps with the ground-truth "gold" summary.
-
Negative Information ($z_n$): Content present in the candidate but absent from the gold summary. To ensure these signals don't mix, they use an informativeness objective (to make the variables predictive) and an adversarial objective (to ensure $z_p$ doesn't contain $z_n$ info, and vice versa).
-
Information Augmented Decoding: The decoder is modified in two ways. First, it concatenates the candidate representations with the source document representations. Second, it "injects" the squeezed positive and negative latent variables directly into the cross-attention layer, effectively telling the decoder: "Focus more on these concepts, and suppress these others."
Datasets
The researchers evaluated DIR on three large-scale MDSS benchmarks:
- Multi-Xscience: A dataset of over 40,000 instances where the goal is to summarize a query paper's abstract and its cited references into a "Related Work" paragraph.
- TAD: A multi-field dataset collected from S2ORC.
- TAS2: A computer science-focused dataset collected from the Delve corpus.
Key Insights and Ideas
-
Candidates as Dual Signals: A summary candidate isn't just "good" or "bad." It's a mixture of helpful matches and misleading mismatches. By treating mismatches as negative signals, the model learns what to avoid, which is just as important for quality as knowing what to include. Figure 1 illustrates how a candidate can have both matched (positive) and mismatched (negative) content compared to a gold summary.
-
The Power of Ranking: Simply concatenating all available candidates can actually hurt performance because it dilutes the role of the source papers. Selecting the top 3 candidates ($K=3$) was found to be the "sweet spot" for performance.
- LLMs as Cheap Teachers: While training models like BART to generate candidates is effective, using a Large Language Model (like GPT-3.5) to generate candidates via prompting is a highly efficient way to get high-quality external guidance without heavy retraining.
Key Results
The DIR framework consistently outperformed base models across all datasets and metrics (ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore).
- New SOTA: The combination of KGSum + DIR achieved a new state-of-the-art performance on all three datasets, surpassing the previous leader, EDITSum.
-
Human Preference: In blind tests, human evaluators significantly preferred summaries generated by DIR-equipped models. For the TransS2S model, the DIR version was preferred 59.1% of the time, compared to only 21.1% for the base model. Human evaluation results showing a clear preference for DIR-equipped models across different architectures.
-
Disentanglement Success: Visualization of the latent space (using t-SNE) confirmed that the positive and negative information were successfully separated into distinct clusters after training.
Limitations and Future Work
- Computational Overhead: Generating multiple candidates and ranking them adds a second stage to the inference process, which increases computational cost and latency.
- Candidate Quality Dependency: The framework's success relies on the quality of the first-stage models. If all candidates are extremely poor, the "positive" signal may be too weak to be useful.
- Future Directions: The authors suggest exploring more diverse ways to generate candidates and investigating how this "positive/negative" disentanglement might apply to other conditional text generation tasks beyond scientific summarization.
Summary updated