Introduction: The Challenge of Summarization Evaluation
Evaluating automatic text summarization is a persistent challenge in natural language processing. Standard automatic metrics, such as ROUGE, rely on measuring n-gram overlap between a system-generated summary and a single human-written "gold" reference. However, this approach suffers from reference bias: a system might capture perfectly valid salient points that happen to be absent from that specific reference, resulting in an unfairly low score.
While manual human evaluation is an alternative, it is often plagued by inconsistency. Non-expert readers frequently struggle to determine which information is truly "important" in a long article, leading to low inter-annotator agreement. HighRES (HIGHlight-based REference-less Summarization) addresses these issues by providing a scalable, crowd-sourcing-friendly framework that replaces biased single references with consensus-based document highlights.
The HighRES Framework
The HighRES framework shifts the focus from comparing summaries to other summaries to comparing summaries directly to the source document's salient content. The process consists of four distinct stages:
- Highlight Annotation: Multiple crowd-workers highlight words or phrases in a source document they consider salient, constrained by a word budget. This creates a "heat map" where darker text represents a higher consensus on importance.
- Content Evaluation: Judges are presented with the highlighted source document and a system summary. They rate the summary on two 1-100 scales: Recall (does it contain all the important info?) and Precision (does it contain only important info?).
- Quality Evaluation: Fluency (grammatical correctness) and Clarity (the ease of identifying referents) are rated as independent metrics.
- HROUGE: A novel "Highlight-based ROUGE" metric is introduced. Unlike standard ROUGE, HROUGE compares the system summary to the source document, weighting n-grams based on the frequency with which they were highlighted by humans.
Experimental Design and Datasets
The researchers benchmarked HighRES using the XSUM (Extreme Summarization) dataset, which contains BBC articles paired with highly abstractive, single-sentence summaries. XSUM was chosen as a "stress test" because its summaries are not simple extracts from the source. The study sampled 50 articles and collected 10 human highlight sets for each to establish a robust evaluation baseline.
Key Insights and Results
A primary advantage of the HighRES approach is the reduction of cognitive load for judges. By providing a pre-highlighted "heat map," judges do not need to determine saliency on the fly, which significantly improves inter-annotator agreement and reduces variance compared to standard reference-based methods.
Key findings from the evaluation include:
- Clarity vs. Fluency: The authors identified a low correlation (0.68) between fluency and clarity. A summary can be perfectly grammatical yet confusing if it lacks context for its subjects, proving the necessity of measuring these traits separately.
- System Differentiation: HighRES proved more effective at exposing performance gaps between models. It revealed that Topic-aware Convolutional networks (TCONVS2S) were significantly better at capturing high-level document meaning than Pointer-Generator networks—a distinction less apparent under traditional metrics.
- Breaking the Reference Chain: By evaluating against source highlights, HighRES avoids penalizing systems for valid content selection that differs from a single human reference.
Conclusion and Future Directions
HighRES provides a more nuanced and reliable look at summary quality without being tethered to a single human example. While HROUGE still relies on n-gram matching and may struggle with complex paraphrasing, the framework offers a significant step toward more objective manual evaluation. Future work involves extending HighRES to multi-document summarization and exploring the use of human highlights as a saliency signal to train more effective summarization models.
Summary updated