Abstract
Traditional evaluation of text summarization relies on human-written "gold standard" summaries, which are expensive and subjective. This paper presents a fully automatic, model-free method for content selection evaluation. By comparing the word distribution of a summary directly to its source text using Jensen-Shannon (JS) divergence, the authors achieve a correlation of up to 0.88 with manual expert evaluations. These results suggest that distributional similarity is a powerful proxy for content importance, enabling large-scale evaluation without the need for human models.
Introduction: The Challenge of Model-Based Evaluation
In Natural Language Processing (NLP), the standard metric for summarization is ROUGE, which measures the overlap between a machine-generated summary and human-written models. However, this approach faces significant hurdles:
- Cost and Scalability: Producing high-quality human summaries for large datasets is prohibitively expensive.
- Subjectivity: Human writers often disagree on what content is most important, leading to potential bias in evaluation.
- Static Nature: Model-based metrics cannot easily adapt to new datasets where no human ground truth exists.
The authors propose a "model-free" alternative: evaluating a summary based solely on its relationship with the source documents.
Methodology: Distributional Similarity
The core hypothesis is that an effective summary should mirror the word distribution of the input text. The authors explored several features to quantify this relationship:
- Distributional Similarity Measures: Information-theoretic metrics like Kullback-Leibler (KL) Divergence and Jensen-Shannon (JS) Divergence were used to measure the distance between the probability distributions of words in the input and the summary.
- Topic Signatures: The authors identified "topic words"—terms with high descriptive power in the source—and calculated the percentage of these words included in the summary.
- Summary Likelihood: Probability models (unigram and multinomial) were used to estimate how likely the summary was to have been generated from the source text.
Key preprocessing steps, such as stemming and stop-word filtering, were found to be critical for the success of these metrics.
Experimental Setup
The study used data from the TAC 2008 (Text Analysis Conference) summarization track, encompassing 48 topics and 57 different summarization systems. The automatic model-free scores were validated against two gold-standard manual evaluations:
- Pyramid Scores: A robust human evaluation of content coverage.
- Responsiveness: A holistic human judgment of how well a summary meets a user's information needs.
Results: The Dominance of JS Divergence
The experiments revealed that JS Divergence is the most effective model-free metric. Unlike KL divergence, JS divergence is symmetric and always defined, providing a stable measure for comparing short summaries to long source documents.
As shown in Figure 2, JS Divergence achieved a 0.88 correlation with manual Pyramid scores for query-focused summaries. This performance is remarkably close to ROUGE-2 (0.90–0.94), which requires human models. Notably, a linear regression model combining all features (0.867) did not outperform JS Divergence alone, highlighting the strength of this single metric.
The method also proved effective for "update" summarization tasks, where the goal is to summarize new information given a background context.
Figure 4 demonstrates that for update summaries, comparing the summary to the "update input only" yields higher correlations than including background information, suggesting that the metrics are sensitive to the specific content goals of the task.
Discussion: Macro vs. Micro Evaluation
While the model-free approach is highly effective at the macro level (ranking different summarization systems), it remains less reliable at the micro level (evaluating a specific summary for a specific document). This suggests that while general word distribution is a strong proxy for system quality, individual document nuances still pose a challenge.
Furthermore, the authors emphasize that these metrics focus exclusively on content selection. They do not account for linguistic quality, such as grammar, coherence, or syntax. A summary consisting of a "word salad" of important keywords would score highly on JS Divergence but would be unreadable to a human reader.
Conclusion
This work demonstrates that model-free evaluation is a viable and powerful alternative to traditional model-based metrics. By leveraging JS Divergence to compare summaries with source texts, researchers can rank summarization systems with high accuracy without the need for expensive human-written models. Future work will focus on integrating linguistic quality assessments and refining metrics to better capture query-specific relevance.
Summary updated