Automated text summarization has advanced rapidly, yet evaluating these summaries remains a significant challenge. Traditional metrics like ROUGE rely on human-written reference summaries, which are expensive to produce and often fail to capture semantic nuances or grammatical correctness. This paper, presented at AAAI-22, introduces a novel, reference-free approach by reviving a classic information-theoretic concept: the Shannon Game.
The Shannon Game Framework
Originally proposed in the 1950s, the Shannon Game involves guessing a document letter-by-letter. A "good" summary is one that provides enough context to significantly reduce the number of guesses required. The authors modernize this concept by replacing human guessers with Autoregressive Language Models (e.g., GPT-2).
They propose two primary metrics:
- Information Difference ($ID$): Measures the "surprisal" (negative log-likelihood) of a document. It calculates the difference between the information needed to generate the document from scratch versus generating it after being prompted with the summary.
- Shannon Score ($s$): A normalized ratio representing the help provided by the summary relative to the maximum possible help (providing the document itself as a prompt).
To accommodate long documents, the researchers employ a sentence-independence assumption, using the summary to predict each sentence of the source document individually.
Experimental Results
The metrics were benchmarked using SummEval, which contains summaries from 17 different models scored by human experts across four dimensions: Coherence, Consistency, Fluency, and Relevance.
As shown in the results, the Shannon-based metrics achieved state-of-the-art correlation with human judgments for Coherence (0.47) and Relevance (0.66). Notably, Information Difference significantly outperformed ROUGE-1 across all categories without requiring any reference summaries. The metrics were also uniquely versatile, ranking in the top 5 across all four human-judged categories.
Sensitivity and Syntax
A key advantage of these metrics over embedding-based approaches is their sensitivity to syntax and word order. Unlike metrics that only look at semantic overlap, the Shannon Score is highly sensitive to the structure of the summary.
Validation on the CNN/DailyMail dataset demonstrated that when words in a summary are shuffled, the scores drop significantly. This occurs because a disorganized prompt provides poor context for the language model, increasing the surprisal of the source document. This sensitivity allows the metrics to penalize ungrammatical or nonsensical summaries that might otherwise fool keyword-matching systems.
Conclusion and Future Work
The Shannon Score and Information Difference provide a robust, reference-free alternative to traditional summarization metrics. While they exhibit some length bias and higher computational costs than ROUGE, their high correlation with human judgment makes them powerful tools for evaluation. Future research may involve using these metrics as reward signals for reinforcement learning to train summarization models directly, potentially bypassing the need for human-labeled data entirely.
Summary updated