Abstract
Traditional evaluation of extractive summarization often treats systems as "black boxes," conflating the quality of the scoring model with the efficiency of the search algorithm. This paper introduces a principled framework that decomposes summarizers into an objective function ($\theta$) and an optimizer ($O$). By isolating these components, we evaluate how well current mathematical models of summary quality correlate with human judgments across standard benchmarks. Our findings reveal that even top-performing systems rely on objective functions that correlate poorly with human intuition, highlighting a significant gap in our understanding of automated summary quality.
Introduction
In Natural Language Processing (NLP), the standard approach to evaluating summarization systems involves comparing a system's output to human-written references using metrics like ROUGE. However, this end-to-end evaluation fails to identify whether a poor summary is the result of a flawed quality model or an ineffective search strategy. Furthermore, optimizing for ROUGE can lead to systems that "game" the metric without capturing the nuances of human-perceived quality. This research proposes a formal decomposition to better understand the internal mechanics of extractive summarizers.
Methodology: The $(\theta, O)$ Framework
The authors propose that any extractive summarizer can be formally defined as $\sigma(D) = O(\theta, D)$, where:
- $\theta$ (Objective Function): A function that assigns a quality score to a candidate summary.
- $O$ (Optimizer): The algorithm (e.g., greedy search, Integer Linear Programming) used to find the summary $S$ that maximizes $\theta$.
To test this framework, the researchers isolated the implicit objective functions from several prominent summarization models, including ICSI (bigram coverage), LexRank (graph-based centrality), LSA (Latent Semantic Analysis), and KL-Greedy (divergence minimization). These functions were then used to rank thousands of candidate summaries from the Text Analysis Conference (TAC) datasets.
Experimental Setup
The study utilized the TAC-2008 and TAC-2009 benchmarks, which contain human-assigned scores for "Pyramid" (content coverage) and "Responsiveness" (overall quality) for hundreds of system-generated summaries. The performance of each objective function was measured using Pearson’s $r$, Spearman’s $\rho$, and Normalized Discounted Cumulative Gain (NDCG) to assess their correlation with human judgments.
Results and Discussion
The analysis revealed a surprising discrepancy between end-to-end performance and objective function quality. While systems like ICSI are often top-tier in ROUGE-based evaluations, their underlying objective functions show relatively low correlation with human scores.
As shown in the table above, LexRank and LSA objective functions demonstrated the strongest correlations with human Pyramid scores in TAC-2009 ($r \approx 0.37$), significantly outperforming the bigram-centric ICSI model ($r \approx 0.13$). However, even these "best" functions only achieve modest correlation levels. In contrast, the correlation between different human metrics (Pyramid vs. Responsiveness) remains high at approximately $0.7$, serving as an upper bound that current automated models fail to approach.
Key Insights
- The Search-Quality Gap: Systems may perform well in standard evaluations due to efficient optimization or "lucky" side effects, even if their internal model of quality is primitive.
- Ranking the Best: NDCG scores suggest that while some models struggle to rank mediocre summaries, they are more effective at identifying the highest-quality candidates, which explains their success in traditional top-1 evaluations.
- The Need for a Universal $\theta$: The modest correlations (0.2 to 0.3 range) indicate that our mathematical definitions of summary quality are still far from capturing human intuition.
Conclusion and Future Work
This framework provides a new lens for analyzing summarization systems by separating the "what" (objective) from the "how" (optimization). While currently limited to extractive summarization, the findings suggest that the path forward involves developing a more robust, "universal" objective function that truly reflects human notions of quality. Such a function could then be paired with powerful optimizers to generate summaries that are both mathematically optimal and human-centric.
Summary updated