A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments

Maxime Peyrard, Judith Eckle-KohlerACL 2017 · DOI

Abstract

Traditional evaluation of extractive summarization often treats systems as "black boxes," conflating the quality of the scoring model with the efficiency of the search algorithm. This paper introduces a principled framework that decomposes summarizers into an objective function ($\theta$) and an optimizer ($O$). By isolating these components, we evaluate how well current mathematical models of summary quality correlate with human judgments across standard benchmarks. Our findings reveal that even top-performing systems rely on objective functions that correlate poorly with human intuition, highlighting a significant gap in our understanding of automated summary quality.

Introduction

In Natural Language Processing (NLP), the standard approach to evaluating summarization systems involves comparing a system's output to human-written references using metrics like ROUGE. However, this end-to-end evaluation fails to identify whether a poor summary is the result of a flawed quality model or an ineffective search strategy. Furthermore, optimizing for ROUGE can lead to systems that "game" the metric without capturing the nuances of human-perceived quality. This research proposes a formal decomposition to better understand the internal mechanics of extractive summarizers.

Methodology: The $(\theta, O)$ Framework

The authors propose that any extractive summarizer can be formally defined as $\sigma(D) = O(\theta, D)$, where:

  • $\theta$ (Objective Function): A function that assigns a quality score to a candidate summary.
  • $O$ (Optimizer): The algorithm (e.g., greedy search, Integer Linear Programming) used to find the summary $S$ that maximizes $\theta$.

To test this framework, the researchers isolated the implicit objective functions from several prominent summarization models, including ICSI (bigram coverage), LexRank (graph-based centrality), LSA (Latent Semantic Analysis), and KL-Greedy (divergence minimization). These functions were then used to rank thousands of candidate summaries from the Text Analysis Conference (TAC) datasets.

Experimental Setup

The study utilized the TAC-2008 and TAC-2009 benchmarks, which contain human-assigned scores for "Pyramid" (content coverage) and "Responsiveness" (overall quality) for hundreds of system-generated summaries. The performance of each objective function was measured using Pearson’s $r$, Spearman’s $\rho$, and Normalized Discounted Cumulative Gain (NDCG) to assess their correlation with human judgments.

Results and Discussion

The analysis revealed a surprising discrepancy between end-to-end performance and objective function quality. While systems like ICSI are often top-tier in ROUGE-based evaluations, their underlying objective functions show relatively low correlation with human scores.

Correlation of various objective functions (θ) with human judgments (Responsiveness and Pyramid) on TAC-2008 and TAC-2009 datasets.

As shown in the table above, LexRank and LSA objective functions demonstrated the strongest correlations with human Pyramid scores in TAC-2009 ($r \approx 0.37$), significantly outperforming the bigram-centric ICSI model ($r \approx 0.13$). However, even these "best" functions only achieve modest correlation levels. In contrast, the correlation between different human metrics (Pyramid vs. Responsiveness) remains high at approximately $0.7$, serving as an upper bound that current automated models fail to approach.

Key Insights

  • The Search-Quality Gap: Systems may perform well in standard evaluations due to efficient optimization or "lucky" side effects, even if their internal model of quality is primitive.
  • Ranking the Best: NDCG scores suggest that while some models struggle to rank mediocre summaries, they are more effective at identifying the highest-quality candidates, which explains their success in traditional top-1 evaluations.
  • The Need for a Universal $\theta$: The modest correlations (0.2 to 0.3 range) indicate that our mathematical definitions of summary quality are still far from capturing human intuition.

Conclusion and Future Work

This framework provides a new lens for analyzing summarization systems by separating the "what" (objective) from the "how" (optimization). While currently limited to extractive summarization, the findings suggest that the path forward involves developing a more robust, "universal" objective function that truly reflects human notions of quality. Such a function could then be paired with powerful optimizers to generate summaries that are both mathematically optimal and human-centric.

Summary updated

BibTeX
@inproceedings{peyrard2017principled,
  author = {Maxime Peyrard and Judith Eckle-Kohler},
  title = {{A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.18653/v1/p17-2005},
}
Figure