TL;DR
This paper challenges the long-held assumption that the best automated summary evaluation metrics are those that most closely correlate with human judgments. The authors demonstrate how this reliance on human scores can lead to "false improvements" where metrics are optimized for annotator biases rather than true summary quality. They propose a novel "max-help" criterion that selects metrics based on their ability to extract information from summaries, independent of human scores, and show this criterion is robust across diverse datasets.
What Problem Does This Paper Address?
Evaluating the quality of automatically generated text summaries is a critical task in Natural Language Processing. Traditionally, the gold standard for evaluating automated metrics has been their correlation with human judgments. The assumption is that a metric is "good" if it assigns scores similar to what humans would. However, this paper questions this paradigm, arguing that human annotators themselves can exhibit biases (e.g., favoring summaries that are easier to score or focus on specific parts of the text). This reliance on potentially biased human scores can lead to metrics that appear to improve but don't necessarily reflect genuine summary quality. The authors aim to identify a more reliable, human-free criterion for selecting optimal evaluation metrics.
Methods
The core of the paper's methodology involves analyzing the BLANC (a BERT-based metric that uses a cloze task to assess how well a summary helps predict masked tokens in the original text) evaluation metric and its variants.
- False Improvement Demonstration: The authors introduce a "limited-text BLANC" by restricting the evaluation to only the most similar sentences between the summary and the original text. They show that this modification can artificially increase correlation with human scores by exploiting annotator biases, even if it degrades the metric's actual ability to assess overall summary quality.
- True Improvement Demonstration: They propose a "full BLANC" by combining BLANC-help (using a pre-trained model to read the summary) and BLANC-tune (lightly tuning the model on the summary). This approach aims to genuinely improve the metric's performance by leveraging the summary more effectively.
- Max-Help Criterion: The central novel contribution is the "max-help" criterion. Instead of maximizing correlation with human scores, this criterion selects the metric configuration that achieves the highest average score (i.e., the best at extracting useful information from summaries). This criterion is designed to be human-free and relies on the intrinsic ability of the metric to perform its task.
Datasets
The paper utilizes existing datasets for evaluation:
- SummEval: A dataset of text-summary pairs with human annotations, used to evaluate correlations with human scores.
- CNN/Daily Mail dataset: Used for evaluating the robustness of the proposed criterion on news articles.
- Custom datasets: The authors also create specific text-summary pairs for their toy examples to illustrate specific points, including summaries of "Harry Potter And the Sorcerer's Stone" and news articles.
Key Insights and Ideas
- Human scores can be misleading: The paper strongly suggests that optimizing solely for correlation with human scores can lead to metrics that are "fooled" by annotator biases, rather than genuinely improving summary evaluation.
- Bias towards "easy" summaries: Annotators may favor summaries that are easier to process or focus on specific, salient parts of the text, which can inflate correlation scores for metrics that also favor these characteristics.
- "Max-help" as a robust, human-free criterion: The proposed "max-help" criterion, which selects metrics based on their intrinsic ability to extract information (measured by average score), offers a promising alternative to human-score correlation.
- Robustness across diverse texts: The "max-help" criterion's ability to identify optimal metric configurations that are consistent across very different types of texts and summaries (including short snippets) highlights its potential for broad applicability.
Key Results
- The "limited-text BLANC" demonstrated a higher Spearman correlation with human scores compared to the original BLANC in toy examples, illustrating a "false improvement."
- The "full BLANC" showed legitimate increases in correlation with human scores, suggesting a true improvement in evaluation capability.
- The "max-help" criterion consistently identified the same optimal metric configurations across four different datasets (CNN, Daily Mail, top sentences, random sentences), demonstrating its robustness.
- Figure 4 and 5 show that deviations from the "optimal" setup identified by the "max-help" criterion lead to a drop in the average BLANC score across all datasets, reinforcing the criterion's validity.
Limitations and Future Work
- Focus on BLANC: While the authors use BLANC as a case study, they acknowledge that similar observations and conclusions could be drawn from other question-answering based evaluation measures.
- Human-free criterion applicability: The paper suggests that the "max-help" criterion can be applied without human summaries, potentially by using random sentences from texts, but further validation might be needed for broader applications.
- Future work: The authors imply that exploring this "max-help" criterion with other evaluation metrics and further investigating its robustness across even more diverse domains and summary types would be valuable.
Summary updated