Assessing the Effect of Inconsistent Assessors on Summarization Evaluation

Karolina Owczarzak, Peter A. Rankel, Hoa Trang Dang, John M. ConroyACL 2012

Introduction

In Natural Language Processing (NLP), human judgment remains the "gold standard" for evaluating AI-generated summaries. Researchers typically rely on three primary metrics: Readability (linguistic quality), Responsiveness (content relevance to a prompt), and Pyramid scores (presence of specific information "nuggets"). However, human assessors are inherently inconsistent due to fatigue, subjective bias, and varying interpretations of quality. This study investigates the extent of this inconsistency and its downstream effects on system rankings and the training of automatic evaluation metrics.

Methodology

The researchers utilized data from the Text Analysis Conference (TAC) Summarization tracks (2008–2011). To measure consistency, they identified pairs of identical summaries—instances where different versions of the same summarization system produced the exact same output for a topic—that were evaluated by the same assessor at different times.

Key analytical techniques included:

  • Krippendorff’s Alpha: To quantify reliability across scores for identical summary pairs.
  • Auto-Pyramid Analysis: Using an automated function to check if assessors consistently identified the same Summary Content Units (SCUs) across identical summaries.
  • Impact Simulation: Recalculating system leaderboards after removing the least consistent assessors.
  • Metric Retraining: Training the CLASSY automatic evaluation metric (a regression-based scorer) specifically on "clean" data to observe performance gains.

The Robustness of System Rankings

A primary concern in evaluation is whether human noise invalidates system leaderboards. The study found that while individual judges are often inconsistent—particularly regarding Readability—the aggregate noise tends to average out across multiple topics and assessors.

As shown in the table below, removing the one or two most inconsistent assessors ("worst") resulted in system rankings that remained almost identical to the original rankings. Pearson’s $r$ correlations remained above 0.99, suggesting that current evaluation frameworks are remarkably robust to individual assessor noise.

Correlation of system rankings after removing the most inconsistent assessors.

Improving Automatic Evaluation Metrics

While system rankings are stable, human inconsistency poses a significant challenge for developing automatic metrics like ROUGE or CLASSY. These metrics are often penalized for failing to emulate the random errors or subjective shifts present in human training data.

The researchers demonstrated that "cleaning" the training data by removing inconsistent human judgments consistently improves the predictive power of automated scorers. For instance, training the CLASSY metric on more consistent data from previous years led to measurable performance boosts when evaluated on subsequent datasets.

Performance of the CLASSY metric when trained on "clean" vs. original human data.

As illustrated in the CLASSY results, the "new" models (trained on cleaned data) frequently outperformed the baseline models across Readability, Responsiveness, and Pyramid metrics. Specifically, correlation with human Readability scores on 2011 data improved from 0.768 to 0.793 when the model was trained on more consistent 2009–2010 data.

Conclusion

This research highlights a "Garbage In, Garbage Out" reality for NLP evaluation. While the high-level ranking of summarization systems is resilient to human error, the development of next-generation automatic metrics is hindered by noisy annotations. By identifying and removing inconsistent data, researchers can build more reliable automated tools that better align with high-quality human judgment. Future work may focus on refining Readability guidelines to reduce the inherent subjectivity that makes it the most difficult metric for humans to score consistently.

Summary updated

BibTeX
@inproceedings{owczarzak2012assessing,
  author = {Karolina Owczarzak and Peter A. Rankel and Hoa Trang Dang and John M. Conroy},
  title = {{Assessing the Effect of Inconsistent Assessors on Summarization Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2012},
}
Figure