A Supervised Aggregation Framework for Multi-Document Summarization

Yulong Pei, Wenpeng Yin, Qifeng Fan, Lian'en HuangCOLING 2012 · OpenReview

Abstract

Yulong Pei, Wenpeng Yin, Qifeng Fan, Lian’en Huang. Proceedings of COLING 2012.

2012.

Introduction

Multi-document summarization (MDS) is the complex task of generating a concise, representative summary from a collection of documents focused on the same topic. The core technical challenge in MDS is sentence ranking: determining which sentences within the corpus are the most salient. Traditional approaches generally fall into two categories: single-method models that rely on specific features like word frequency or graph centrality, and unsupervised ensembles that use "wisdom of the crowd" techniques to reach a consensus.

However, single-method models often provide a one-dimensional view of document importance, while unsupervised ensembles can be misled if mediocre models outvote superior ones. This paper introduces a Supervised Summarization Aggregation (SSA) framework designed to overcome these limitations by learning how to optimally combine multiple "expert" summarizers using human-labeled data.

Methodology

The SSA framework does not attempt to create a new ranking algorithm from scratch. Instead, it treats existing, well-established summarizers as experts and learns a weight vector to aggregate their opinions based on human guidance.

The SSA Framework Architecture

The framework integrates four diverse graph-based summarization methods:

  • LexPageRank: Ranks sentences based on eigenvector centrality within a sentence similarity graph.
  • LexHITS: Adapts the HITS algorithm to treat sentences as "hubs" and "authorities."
  • Manifold-ranking: Utilizes the underlying manifold structure of the document data to identify salient sentences.
  • DivRank: Employs a reinforced random walk to balance sentence prestige with diversity.
Architecture of the Supervised Summarization Aggregation (SSA) framework.

Learning via Ranking SVM

To determine the optimal weights for these summarizers, the authors use Ranking SVM. The final score for any given sentence is calculated as a linear combination of the scores provided by the individual experts. To make the training process computationally feasible, the researchers reformulated the standard Ranking SVM—which typically involves $O(N^2)$ pairwise comparisons—into a matrix-based similarity problem. This optimization significantly reduced the number of constraints, allowing the model to scale efficiently.

Experimental Setup

The framework was evaluated using the DUC2004 dataset, a standard benchmark for generic multi-document summarization. The dataset was split into 40 document clusters for training and 10 for testing.

Labeling Strategy

Because DUC2004 provides human-written summaries rather than sentence-level labels, the authors created a proxy labeling scheme. Sentences were scored based on their n-gram overlap (ROUGE-style) with human references and categorized into three classes: +1 (Summary), 0 (Possible Summary), and -1 (Non-summary). A key finding during this phase was that a "maximum" labeling scheme—taking the highest score from any single human reference—performed better than an "average" scheme, as it better preserved the signal of highly salient sentences.

Results and Discussion

The SSA framework demonstrated superior performance across all metrics compared to baselines, individual models, and unsupervised ensemble methods.

Performance Comparison

SSA achieved a ROUGE-1 score of 0.39766, significantly outperforming the best individual component (Manifold-ranking at 0.37809) and the leading unsupervised ensemble (WCS at 0.38227). The results indicate that supervised learning provides the "precise guidance" necessary to identify the most reliable experts for the task.

Key Insights

  • Diversity of Experts: The authors found that the ensemble's strength lies in its diversity. Removing any single component, particularly the Manifold-ranking algorithm, led to a measurable decrease in performance, suggesting that different algorithms capture unique, non-overlapping aspects of sentence importance.
  • Robustness: Compared to unsupervised alternatives, the supervised approach showed narrower confidence intervals and more consistent results across ROUGE-1, ROUGE-2, and ROUGE-SU4 metrics.

Conclusion

The proposed SSA framework proves that supervised aggregation is a powerful tool for multi-document summarization. By leveraging human labels to weight the contributions of various ranking algorithms, the model achieves state-of-the-art results on benchmark datasets. Future research directions include exploring semi-supervised learning to mitigate the cost of human labeling and integrating redundancy removal directly into the supervised objective function.

Summary updated

BibTeX
@inproceedings{pei2012supervised,
  author = {Yulong Pei and Wenpeng Yin and Qifeng Fan and Lian'en Huang},
  title = {{A Supervised Aggregation Framework for Multi-Document Summarization}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=gbA1zzv1iMK},
}
Figure