A Hybrid Hierarchical Model for Multi-Document Summarization

Asli Çelikyilmaz, Dilek Hakkani-TürACL 2010 · OpenReview

Abstract

Scoring sentences in documents given abstract summaries created by humans is important in extractive multi-document summarization. In this paper, we formulate extractive summarization as a two step learning problem building a generative model for pattern discovery and a regression model for inference. We calculate scores for sentences in document clusters based on their latent characteristics using a hierarchical topic model. Then, using these scores, we train a regression model based on the lexical and structural characteristics of the sentences, and use the model to score sentences of new documents to form a summary. Our system advances current state-of-the-art improving ROUGE scores by ~7%. Generated summaries are less redundant and more coherent based upon manual quality evaluations.

Introduction

Multi-document summarization (MDS) aims to distill a cluster of documents into a concise summary. This task faces a significant trade-off between semantic depth and computational efficiency. Supervised methods often struggle with generalization across topics, while unsupervised generative models like Latent Dirichlet Allocation (LDA) can be computationally expensive to re-run for every new document cluster. This paper introduces HybHSum, a hybrid approach that leverages the semantic richness of hierarchical topic models during training to inform a fast, discriminative regression model for efficient inference on new data.

Methodology

The HybHSum framework operates in two distinct phases: a generative phase for pattern discovery and a discriminative phase for efficient scoring.

The Generative Phase: sumHLDA

The authors extend Hierarchical Latent Dirichlet Allocation (hLDA) into a summary-focused version called sumHLDA. Unlike flat topic models, sumHLDA organizes information into a tree structure where the root represents general concepts and leaves represent specific details. By incorporating human-written summaries during training, the model learns to identify paths in the hierarchy that characterize the "summary-worthy" abstractive core of a document cluster.

Hierarchical topic structure and sample path for sentence scoring.

Sentences are scored based on their alignment with these summary-biased paths. This tree-based scoring considers both unigram distributions and topic proportions across the hierarchy levels, ensuring that selected sentences maintain the appropriate level of abstraction.

The Discriminative Phase: Regression for Inference

To enable real-time summarization of new documents, the system avoids re-running the full topic model at test time. Instead, it uses the scores generated by sumHLDA as labels to train a Support Vector Regression (SVR) model. This model utilizes "meta-features"—such as n-gram probabilities relative to the cluster, sentence position, and length—allowing it to generalize the characteristics of high-quality summary sentences to entirely new vocabularies.

Experimental Results

The model was evaluated on standard benchmarks from the Document Understanding Conferences (DUC 2005–2007).

Quantitative Performance

The hierarchical hybrid model, HybHSum₂, achieved a ROUGE-1 score of 45.6, representing a ~7% relative improvement over previous state-of-the-art systems like PYTHY (42.6) and HIERSUM (42.4).

The Importance of Hierarchy

A key finding is that hierarchical modeling allows the system to distinguish between general and specific information more effectively than "flat" models. As shown in the comparison below, the hierarchical model (HybHSum) more closely matches the word frequency distribution of human reference summaries, particularly for specific terms that flat models (HybFSum) often omit entirely.

Comparison of word frequencies between the reference (Ref), the hierarchical model (HybHSum), and a flat model (HybFSum).

Conclusion

HybHSum demonstrates that combining generative hierarchical topic models with discriminative regression provides a powerful and efficient solution for MDS. By "teaching" a regression model the semantic patterns discovered in a hierarchical tree, the system produces summaries that are not only more accurate according to ROUGE metrics but also more coherent and focused according to human evaluation. Future work will explore more sophisticated redundancy elimination and methods to reduce the dependency on human-authored training summaries.

Summary updated

Abstract

Scoring sentences in documents given abstract summaries created by humans is important in extractive multi-document summarization. In this paper, we formulate extractive summarization as a two step learning problem building a generative model for pattern discovery and a regression model for inference. We calculate scores for sentences in document clusters based on their latent characteristics using a hierarchical topic model. Then, using these scores, we train a regression model based on the lexical and structural characteristics of the sentences, and use the model to score sentences of new documents to form a summary. Our system advances current state-of-the-art improving ROUGE scores by ~7%. Generated summaries are less redundant and more coherent based upon manual quality evaluations.
BibTeX
@inproceedings{celikyilmaz2010hybrid,
  author = {Asli Çelikyilmaz and Dilek Hakkani-Tür},
  title = {{A Hybrid Hierarchical Model for Multi-Document Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=HJW2Togu-H},
}
Figure