Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

Chin-Yew Lin, Eduard H. HovyNAACL 2003 · OpenReview

Abstract: Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.

Introduction

In the early 2000s, automated text summarization was experiencing rapid growth, but evaluation remained a significant bottleneck. The gold standard—human judgment—is notoriously slow, expensive, and subjective. For instance, data from the 2001 Document Understanding Conference (DUC) revealed that nearly 18% of evaluations contained conflicting human judgments. Researchers needed a "just-in-time" evaluation method that could reliably and quickly assess system improvements. Inspired by the success of the BLEU metric in machine translation, Chin-Yew Lin and Eduard Hovy investigated whether n-gram co-occurrence statistics could serve as a proxy for human evaluation in summarization.

Methodology

The authors adapted n-gram co-occurrence metrics to the specific requirements of summarization. A key philosophical shift was required: while machine translation evaluation is typically precision-based (ensuring produced words are correct), summarization evaluation is primarily recall-based (ensuring all important information is captured).

The study utilized datasets from DUC 2001 and 2002, which included single-document and multi-document summarization tasks. Human-written "ideal" summaries of varying lengths (50, 100, 200, and 400 words) served as references against outputs from 11–12 automated systems. To validate the automatic metrics, the authors compared them against human scores using Spearman’s rank correlation, measuring how well the metrics mirrored human rankings of the systems.

The Summary Evaluation Environment (SEE) interface used for human evaluation of peer summaries against model references.

The researchers tested several variations, including individual n-grams (unigrams to 4-grams), weighted averages similar to BLEU, and preprocessing steps like stemming and the removal of "stopwords" (common words like "the" or "and").

Key Insights and Results

The most significant finding was the "Unigram Surprise": simple unigram overlap ($Ngram(1,1)$) consistently proved to be the best predictor of human quality scores, often outperforming more complex weighted averages.

The Role of Content vs. Fluency

The authors hypothesized that while longer n-grams (3-grams and 4-grams) are better at measuring grammaticality and fluency, they were less critical for the extractive summarizers of the era. Since these systems pulled existing sentences from source texts, they were inherently grammatical; thus, evaluation primarily needed to measure content coverage, which unigrams do effectively.

Performance Metrics

Unigram overlap achieved remarkably high Spearman correlation coefficients, often ranging between 0.88 and 0.99 across various tasks. While direct application of the BLEU metric showed promise (correlations of 0.66 to 0.82), the modified recall-based unigram approach was significantly more stable across different summary lengths and tasks.

Statistical correlation metrics (Spearman, Pearson, etc.) for various n-gram configurations across different summarization tasks.

Furthermore, the authors found that excluding stopwords significantly improved correlation by forcing the metric to focus on high-information content. They also demonstrated that if the automatic metric indicated a statistically significant difference between two systems, human judges were highly likely to agree, confirming the metric's utility for guiding system development.

Limitations and Future Directions

The authors acknowledged that unigram-based metrics might struggle with abstractive summaries, where a system might use synonyms not present in the human reference. They suggested future work involving weighting n-grams by information content (e.g., tf-idf) or using automated Question-Answering (QA) tests. This research ultimately paved the way for the development of ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which became the industry standard for summarization evaluation for nearly two decades.

Summary updated

Abstract

Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
BibTeX
@inproceedings{lin2003automatic,
  author = {Chin-Yew Lin and Eduard H. Hovy},
  title = {{Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
  url = {https://openreview.net/forum?id=rkNOvXZ_ZB},
}
Figure