Abstract
Until recently, there were no common, convenient, and repeatable evaluation methods that could be easily applied to support the fast turn-around development of automatic text summarization systems. In this paper, we introduce an information-theoretic approach to automatic evaluation of summaries based on the Jensen-Shannon (JS) divergence of distributions between an automatic summary and a set of reference summaries. Several variants of the approach are also considered and compared. The results indicate that the JS divergence-based evaluation method achieves comparable performance with the common automatic evaluation method ROUGE in single-document summarization tasks, while achieving better performance than ROUGE in multi-document summarization tasks.
Introduction
Evaluating the quality of automatically generated text summaries traditionally required expensive and time-consuming human assessment. While automatic metrics like ROUGE (based on n-gram co-occurrence statistics) became the standard to bypass this bottleneck, they largely rely on simple overlap counting. This paper aims to build a more principled, statistical framework based on language modeling and information theory. Specifically, we frame summary evaluation as a problem of estimating the probability distribution of words in a summary and measuring how far the machine-generated distribution diverges from the human-generated (reference) distribution.
Methodology
We model summaries as generative probabilistic models using n-grams (unigrams, bigrams, or trigrams). To evaluate a system summary, we measure the distance between its estimated word distribution and the reference summary's distribution using divergence metrics.
Divergence Metrics and Smoothing
We tested Kullback-Leibler (KL) divergence and Jensen-Shannon (JS) divergence. JS divergence is preferred because it is symmetric and remains finite even when encountering words present in one distribution but not the other. To handle the "zero-frequency" problem (unseen words), we apply Bayes-smoothing using a Dirichlet prior. This technique redistributes a small amount of probability mass to unseen words based on a large background English corpus (TREC AP88-90).
Handling Multiple References
When multiple human reference summaries are available, we use Bayesian inference to update and refine the posterior distribution of the reference model, creating a more robust target distribution for evaluation.
Experimental Results
The proposed method was evaluated using the DUC 2002 dataset, comparing system-generated summaries against human-assigned "coverage scores."
Single-Document Summarization (SD)
For single-document tasks, unigram models proved optimal. Unsmoothed Jensen-Shannon Divergence (JSD) achieved a Pearson correlation of 0.97 and a Spearman correlation of 0.91 with human scores when using multiple references (MR). This performance is comparable to the highest-performing ROUGE variants, which achieved correlations up to 0.99.
Multi-Document Summarization (MD)
In multi-document tasks, bigram models captured necessary context more effectively. The JSD-based measure over bigrams achieved a Pearson correlation of 0.96 and a Spearman correlation of 0.94 for the MD task with multiple references. This significantly outperforms standard ROUGE variants, which typically scored between 0.84 and 0.92 in multi-document settings.
Discussion and Conclusion
Information theory provides a natural and theoretically sound fit for summarization evaluation. By viewing the task as measuring the distance between probability distributions, we move beyond simple n-gram counting. Our results show that JS divergence is a robust alternative to ROUGE, particularly in complex multi-document scenarios.
However, like ROUGE, this method remains lexically rigid and cannot recognize synonyms or paraphrases. Future work will focus on incorporating semantic equivalents through techniques like query expansion and dependency language models to further improve the accuracy of automatic summary evaluation.
Summary updated