Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation

Nicholas Egan, Oleg V. Vasilyev, John BohannonAAAI 2022 · DOI

Automated text summarization has advanced rapidly, yet evaluating these summaries remains a significant challenge. Traditional metrics like ROUGE rely on human-written reference summaries, which are expensive to produce and often fail to capture semantic nuances or grammatical correctness. This paper, presented at AAAI-22, introduces a novel, reference-free approach by reviving a classic information-theoretic concept: the Shannon Game.

The Shannon Game Framework

Originally proposed in the 1950s, the Shannon Game involves guessing a document letter-by-letter. A "good" summary is one that provides enough context to significantly reduce the number of guesses required. The authors modernize this concept by replacing human guessers with Autoregressive Language Models (e.g., GPT-2).

They propose two primary metrics:

  1. Information Difference ($ID$): Measures the "surprisal" (negative log-likelihood) of a document. It calculates the difference between the information needed to generate the document from scratch versus generating it after being prompted with the summary.
  2. Shannon Score ($s$): A normalized ratio representing the help provided by the summary relative to the maximum possible help (providing the document itself as a prompt).

To accommodate long documents, the researchers employ a sentence-independence assumption, using the summary to predict each sentence of the source document individually.

Experimental Results

The metrics were benchmarked using SummEval, which contains summaries from 17 different models scored by human experts across four dimensions: Coherence, Consistency, Fluency, and Relevance.

Table 1: Correlation of various metrics with human judgments on the SummEval benchmark. Shannon-based metrics achieve state-of-the-art results in Coherence and Relevance.

As shown in the results, the Shannon-based metrics achieved state-of-the-art correlation with human judgments for Coherence (0.47) and Relevance (0.66). Notably, Information Difference significantly outperformed ROUGE-1 across all categories without requiring any reference summaries. The metrics were also uniquely versatile, ranking in the top 5 across all four human-judged categories.

Sensitivity and Syntax

A key advantage of these metrics over embedding-based approaches is their sensitivity to syntax and word order. Unlike metrics that only look at semantic overlap, the Shannon Score is highly sensitive to the structure of the summary.

Figure 1: Distribution of Shannon Score and Information Difference for original, shuffled, and wrong summaries. The metrics clearly distinguish between high-quality summaries and those with corrupted syntax or incorrect content.

Validation on the CNN/DailyMail dataset demonstrated that when words in a summary are shuffled, the scores drop significantly. This occurs because a disorganized prompt provides poor context for the language model, increasing the surprisal of the source document. This sensitivity allows the metrics to penalize ungrammatical or nonsensical summaries that might otherwise fool keyword-matching systems.

Conclusion and Future Work

The Shannon Score and Information Difference provide a robust, reference-free alternative to traditional summarization metrics. While they exhibit some length bias and higher computational costs than ROUGE, their high correlation with human judgment makes them powerful tools for evaluation. Future research may involve using these metrics as reward signals for reinforcement learning to train summarization models directly, potentially bypassing the need for human-labeled data entirely.

Summary updated

BibTeX
@inproceedings{egan2022play,
  author = {Nicholas Egan and Oleg V. Vasilyev and John Bohannon},
  title = {{Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21304},
}
Figure