Abstract
Scoring sentences in documents given abstract summaries created by humans is important in extractive multi-document summarization. In this paper, we formulate extractive summarization as a two step learning problem building a generative model for pattern discovery and a regression model for inference. We calculate scores for sentences in document clusters based on their latent characteristics using a hierarchical topic model. Then, using these scores, we train a regression model based on the lexical and structural characteristics of the sentences, and use the model to score sentences of new documents to form a summary. Our system advances current state-of-the-art improving ROUGE scores by ~7%. Generated summaries are less redundant and more coherent based upon manual quality evaluations.
Introduction
Multi-document summarization (MDS) aims to distill a cluster of documents into a concise summary. This task faces a significant trade-off between semantic depth and computational efficiency. Supervised methods often struggle with generalization across topics, while unsupervised generative models like Latent Dirichlet Allocation (LDA) can be computationally expensive to re-run for every new document cluster. This paper introduces HybHSum, a hybrid approach that leverages the semantic richness of hierarchical topic models during training to inform a fast, discriminative regression model for efficient inference on new data.
Methodology
The HybHSum framework operates in two distinct phases: a generative phase for pattern discovery and a discriminative phase for efficient scoring.
The Generative Phase: sumHLDA
The authors extend Hierarchical Latent Dirichlet Allocation (hLDA) into a summary-focused version called sumHLDA. Unlike flat topic models, sumHLDA organizes information into a tree structure where the root represents general concepts and leaves represent specific details. By incorporating human-written summaries during training, the model learns to identify paths in the hierarchy that characterize the "summary-worthy" abstractive core of a document cluster.
Sentences are scored based on their alignment with these summary-biased paths. This tree-based scoring considers both unigram distributions and topic proportions across the hierarchy levels, ensuring that selected sentences maintain the appropriate level of abstraction.
The Discriminative Phase: Regression for Inference
To enable real-time summarization of new documents, the system avoids re-running the full topic model at test time. Instead, it uses the scores generated by sumHLDA as labels to train a Support Vector Regression (SVR) model. This model utilizes "meta-features"—such as n-gram probabilities relative to the cluster, sentence position, and length—allowing it to generalize the characteristics of high-quality summary sentences to entirely new vocabularies.
Experimental Results
The model was evaluated on standard benchmarks from the Document Understanding Conferences (DUC 2005–2007).
Quantitative Performance
The hierarchical hybrid model, HybHSum₂, achieved a ROUGE-1 score of 45.6, representing a ~7% relative improvement over previous state-of-the-art systems like PYTHY (42.6) and HIERSUM (42.4).
The Importance of Hierarchy
A key finding is that hierarchical modeling allows the system to distinguish between general and specific information more effectively than "flat" models. As shown in the comparison below, the hierarchical model (HybHSum) more closely matches the word frequency distribution of human reference summaries, particularly for specific terms that flat models (HybFSum) often omit entirely.
Conclusion
HybHSum demonstrates that combining generative hierarchical topic models with discriminative regression provides a powerful and efficient solution for MDS. By "teaching" a regression model the semantic patterns discovered in a hierarchical tree, the system produces summaries that are not only more accurate according to ROUGE metrics but also more coherent and focused according to human evaluation. Future work will explore more sophisticated redundancy elimination and methods to reduce the dependency on human-authored training summaries.
Summary updated