HighRES: Highlight-based Reference-less Evaluation of Summarization

Hardy, Shashi Narayan, Andreas Vlachos

Introduction: The Challenge of Summarization Evaluation

Evaluating automatic text summarization is a persistent challenge in natural language processing. Standard automatic metrics, such as ROUGE, rely on measuring n-gram overlap between a system-generated summary and a single human-written "gold" reference. However, this approach suffers from reference bias: a system might capture perfectly valid salient points that happen to be absent from that specific reference, resulting in an unfairly low score.

While manual human evaluation is an alternative, it is often plagued by inconsistency. Non-expert readers frequently struggle to determine which information is truly "important" in a long article, leading to low inter-annotator agreement. HighRES (HIGHlight-based REference-less Summarization) addresses these issues by providing a scalable, crowd-sourcing-friendly framework that replaces biased single references with consensus-based document highlights.

The HighRES Framework

The HighRES framework shifts the focus from comparing summaries to other summaries to comparing summaries directly to the source document's salient content. The process consists of four distinct stages:

  1. Highlight Annotation: Multiple crowd-workers highlight words or phrases in a source document they consider salient, constrained by a word budget. This creates a "heat map" where darker text represents a higher consensus on importance.
  2. Content Evaluation: Judges are presented with the highlighted source document and a system summary. They rate the summary on two 1-100 scales: Recall (does it contain all the important info?) and Precision (does it contain only important info?).
  3. Quality Evaluation: Fluency (grammatical correctness) and Clarity (the ease of identifying referents) are rated as independent metrics.
  4. HROUGE: A novel "Highlight-based ROUGE" metric is introduced. Unlike standard ROUGE, HROUGE compares the system summary to the source document, weighting n-grams based on the frequency with which they were highlighted by humans.
The HighRES assessment interface showing the highlighted source document and evaluation sliders.

Experimental Design and Datasets

The researchers benchmarked HighRES using the XSUM (Extreme Summarization) dataset, which contains BBC articles paired with highly abstractive, single-sentence summaries. XSUM was chosen as a "stress test" because its summaries are not simple extracts from the source. The study sampled 50 articles and collected 10 human highlight sets for each to establish a robust evaluation baseline.

Key Insights and Results

A primary advantage of the HighRES approach is the reduction of cognitive load for judges. By providing a pre-highlighted "heat map," judges do not need to determine saliency on the fly, which significantly improves inter-annotator agreement and reduces variance compared to standard reference-based methods.

Comparison of HighRES against existing summarization evaluation frameworks.

Key findings from the evaluation include:

  • Clarity vs. Fluency: The authors identified a low correlation (0.68) between fluency and clarity. A summary can be perfectly grammatical yet confusing if it lacks context for its subjects, proving the necessity of measuring these traits separately.
  • System Differentiation: HighRES proved more effective at exposing performance gaps between models. It revealed that Topic-aware Convolutional networks (TCONVS2S) were significantly better at capturing high-level document meaning than Pointer-Generator networks—a distinction less apparent under traditional metrics.
  • Breaking the Reference Chain: By evaluating against source highlights, HighRES avoids penalizing systems for valid content selection that differs from a single human reference.

Conclusion and Future Directions

HighRES provides a more nuanced and reliable look at summary quality without being tethered to a single human example. While HROUGE still relies on n-gram matching and may struggle with complex paraphrasing, the framework offers a significant step toward more objective manual evaluation. Future work involves extending HighRES to multi-document summarization and exploring the use of human highlights as a saliency signal to train more effective summarization models.

Summary updated

Abstract

There has been substantial progress in summarization research enabled by the availability of novel, often large-scale, datasets and recent advances on neural network-based approaches. However, manual evaluation of the system generated summaries is inconsistent due to the difficulty the task poses to human non-expert readers. To address this issue, we propose a novel approach for manual evaluation, Highlight-based Reference-less Evaluation of Summarization (HighRES), in which summaries are assessed by multiple annotators against the source document via manually highlighted salient content in the latter. Thus summary assessment on the source document by human judges is facilitated, while the highlights can be used for evaluating multiple systems. To validate our approach we employ crowd-workers to augment with highlights a recently proposed dataset and compare two state-of-the-art systems. We demonstrate that HighRES improves inter-annotator agreement in comparison to using the source document directly, while they help emphasize differences among systems that would be ignored under other evaluation approaches.
BibTeX
@inproceedings{hardy2019highres,
  author = {Hardy and Shashi Narayan and Andreas Vlachos},
  title = {{HighRES: Highlight-based Reference-less Evaluation of Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1330},
}
Figure