TL;DR
This paper introduces a novel hybrid approach to multi-document summarization that simultaneously generates a human-readable summary and extracts supporting key evidence. By employing a "salient representation learning" technique within an enhanced transformer architecture, the system effectively bridges the gap between abstractive summarization and extractive evidence identification, offering a more transparent and trustworthy output.
What Problem Does This Paper Address?
Multi-document summarization (MDS) aims to distill key information from multiple sources on the same topic. While existing methods fall into two camps – extractive (selecting existing sentences) and abstractive (generating new sentences) – there's a growing demand for systems that can do both. Specifically, enterprise users often require not just a concise summary but also the specific pieces of evidence from the source documents that justify the summary's claims. Current abstractive systems, while producing fluent summaries, often lack this crucial explainability, and extractive systems, while providing evidence, can be less coherent. This paper tackles this gap by proposing a unified framework that excels at both tasks, providing a more comprehensive and trustworthy summarization solution.
Methods
The core of the proposed system is a hybrid summarization approach built upon an enhanced encoder-decoder transformer architecture, specifically leveraging the Longformer-Encoder-Decoder (LED) for its ability to handle long sequences. The novelty lies in the joint salient representation learning and multi-task training.
- Salient Representation Learning: This module is designed to identify and integrate "noteworthy features" from the input documents. It involves:
- Sentence Salience Prediction: A feed-forward layer predicts salience scores for each candidate sentence.
- Salience Attention: These scores are used to compute attention weights, creating a global salience vector that aggregates sentence-level importance.
- Dynamic Salience Integration: A gating mechanism dynamically injects this global salience information into the decoder during summary generation. This allows the decoder to selectively leverage the most relevant salient features, improving both summary quality and alignment with extracted evidence.
- Multi-Task Learning: The system is trained to optimize a composite loss function that jointly minimizes:
- Abstractive Loss: Standard cross-entropy loss for generating the summary.
- Extraction Loss: A loss (e.g., Mean Squared Error) for predicting the salience scores of sentences.
This joint training encourages the model to learn representations that are beneficial for both tasks simultaneously. The authors also describe a weakly supervised method for generating pseudo-salience labels, combining lexical coverage (ROUGE scores) and factuality measures (NER precision) to create robust training signals.
Datasets
The paper evaluates the proposed system on two domains:
- News Domain: The MULTI-NEWS dataset, which consists of human-written summaries of news articles, sourced from newser.com.
- Legal Domain: The MULTI-LEXSUM dataset, which contains expert-authored summaries of legal case documents, distributed into three subsets (tiny, short, long) based on source document length.
The paper provides detailed statistics for these datasets, including document/sentence/word counts and source-to-target coverage ratios.
Key Insights and Ideas
- Bridging the Extractive-Abstractive Divide: The most significant insight is the successful integration of evidence extraction and abstractive summarization within a single, coherent framework. This addresses a key limitation of existing methods.
- Salience as a Unifying Signal: The concept of "salience" acts as a crucial link. By learning to identify salient sentences, the model can both extract them as evidence and use their representations to inform the abstractive generation process.
- Explainability through Evidence: The system's ability to provide extracted evidence alongside the generated summary significantly enhances transparency and trustworthiness, a critical factor for real-world applications.
- Efficiency of the Hybrid Approach: Despite performing two tasks, the authors demonstrate that their hybrid approach is computationally efficient, not requiring excessive additional resources compared to baseline models.
Key Results
The proposed MHS-SRL (Multi-doc Hybrid Summarization via Salient Representation Learning) system consistently outperformed baseline models like BART and PRIMERA across various metrics on both news and legal domains.
- News Domain: MHS-SRL achieved the best ROUGE scores and BERTScore, indicating superior lexical coverage, semantic similarity, and fluency compared to baselines. For instance, it achieved an ROUGE-1 F1 score of 49.81, outperforming BART (49.75) and PRIMERA (47.53).
- Legal Domain: The system also showed strong performance, achieving the best ROUGE-1 F1 scores across different subsets of the MULTI-LEXSUM dataset. For the D→T subset, MHS-SRL achieved 47.72, surpassing BART (48.13) and PRIMERA (53.52) in some instances, and for D→S, it achieved 47.72, outperforming BART (43.60) and PRIMERA (46.88).
- Ablation Studies: Removing the salient representation learning component led to a noticeable drop in ROUGE scores (4.26%) and BERTScore (3.15%), underscoring the importance of this module.
- Qualitative Evaluation: Examples provided in the paper (Tables 4, 9, and 10) demonstrate that the extracted evidence is well-aligned with the generated summaries, showcasing the system's ability to provide relevant supporting information.
Limitations and Future Work
- Weakly Supervised Labels: While effective, the weakly supervised method for generating salience scores might not perfectly capture true human judgment. The paper acknowledges that more robust salience labels could further improve performance.
- Future Work: The authors suggest exploring more advanced methods for constructing salience labels and potentially extending the hybrid approach to other domains or summarization tasks. The current work focuses on news and legal domains, and its applicability to other text types could be investigated.
Summary updated