An Information-Theoretic Approach to Automatic Evaluation of Summaries

Chin-Yew Lin, Guihong Cao, Jianfeng Gao, Jian-Yun NieNAACL 2006 · OpenReview

Abstract

Until recently, there were no common, convenient, and repeatable evaluation methods that could be easily applied to support the fast turn-around development of automatic text summarization systems. In this paper, we introduce an information-theoretic approach to automatic evaluation of summaries based on the Jensen-Shannon (JS) divergence of distributions between an automatic summary and a set of reference summaries. Several variants of the approach are also considered and compared. The results indicate that the JS divergence-based evaluation method achieves comparable performance with the common automatic evaluation method ROUGE in single-document summarization tasks, while achieving better performance than ROUGE in multi-document summarization tasks.

Introduction

Evaluating the quality of automatically generated text summaries traditionally required expensive and time-consuming human assessment. While automatic metrics like ROUGE (based on n-gram co-occurrence statistics) became the standard to bypass this bottleneck, they largely rely on simple overlap counting. This paper aims to build a more principled, statistical framework based on language modeling and information theory. Specifically, we frame summary evaluation as a problem of estimating the probability distribution of words in a summary and measuring how far the machine-generated distribution diverges from the human-generated (reference) distribution.

Methodology

We model summaries as generative probabilistic models using n-grams (unigrams, bigrams, or trigrams). To evaluate a system summary, we measure the distance between its estimated word distribution and the reference summary's distribution using divergence metrics.

Divergence Metrics and Smoothing

We tested Kullback-Leibler (KL) divergence and Jensen-Shannon (JS) divergence. JS divergence is preferred because it is symmetric and remains finite even when encountering words present in one distribution but not the other. To handle the "zero-frequency" problem (unseen words), we apply Bayes-smoothing using a Dirichlet prior. This technique redistributes a small amount of probability mass to unseen words based on a large background English corpus (TREC AP88-90).

Handling Multiple References

When multiple human reference summaries are available, we use Bayesian inference to update and refine the posterior distribution of the reference model, creating a more robust target distribution for evaluation.

Experimental Results

The proposed method was evaluated using the DUC 2002 dataset, comparing system-generated summaries against human-assigned "coverage scores."

Single-Document Summarization (SD)

For single-document tasks, unigram models proved optimal. Unsmoothed Jensen-Shannon Divergence (JSD) achieved a Pearson correlation of 0.97 and a Spearman correlation of 0.91 with human scores when using multiple references (MR). This performance is comparable to the highest-performing ROUGE variants, which achieved correlations up to 0.99.

Table 1: Pearson (P) and Spearman (S) correlations for Unigram models across different divergence metrics.

Multi-Document Summarization (MD)

In multi-document tasks, bigram models captured necessary context more effectively. The JSD-based measure over bigrams achieved a Pearson correlation of 0.96 and a Spearman correlation of 0.94 for the MD task with multiple references. This significantly outperforms standard ROUGE variants, which typically scored between 0.84 and 0.92 in multi-document settings.

Table 2: Pearson (P) and Spearman (S) correlations for Bigram models. Note the high correlation for JSD in multi-document (MD) tasks with multiple references (MR).

Discussion and Conclusion

Information theory provides a natural and theoretically sound fit for summarization evaluation. By viewing the task as measuring the distance between probability distributions, we move beyond simple n-gram counting. Our results show that JS divergence is a robust alternative to ROUGE, particularly in complex multi-document scenarios.

However, like ROUGE, this method remains lexically rigid and cannot recognize synonyms or paraphrases. Future work will focus on incorporating semantic equivalents through techniques like query expansion and dependency language models to further improve the accuracy of automatic summary evaluation.

Summary updated

Abstract

Until recently there are no common, convenient, and repeatable evaluation methods that could be easily applied to support fast turn-around development of automatic text summarization systems. In this paper, we introduce an information-theoretic approach to automatic evaluation of summaries based on the Jensen-Shannon divergence of distributions between an automatic summary and a set of reference summaries. Several variants of the approach are also considered and compared. The results indicate that JS divergence-based evaluation method achieves comparable performance with the common automatic evaluation method ROUGE in single documents summarization task; while achieves better performance than ROUGE in multiple document summarization task.
BibTeX
@inproceedings{lin2006information,
  author = {Chin-Yew Lin and Guihong Cao and Jianfeng Gao and Jian-Yun Nie},
  title = {{An Information-Theoretic Approach to Automatic Evaluation of Summaries}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
  url = {https://openreview.net/forum?id=Sy-pPX-d-r},
}
Figure