Corroborating Text Evaluation Results with Heterogeneous Measures

Enrique Amigó, Julio Gonzalo, Jesús Giménez, Felisa VerdejoEMNLP 2011

Abstract

In fields like Machine Translation (MT) and Automatic Summarization (AS), researchers often rely on a single metric, such as BLEU or ROUGE, despite their known limitations. This paper proposes a shift toward using a diverse "jury" of metrics. By formalizing the concepts of additive reliability and heterogeneity, the authors demonstrate that corroborating evaluation results with multiple, heterogeneous measures consistently increases the reliability of NLP evaluations. Even metrics with lower individual correlations to human judgments can significantly enhance the overall certainty of an evaluation conclusion if they capture different linguistic properties.

Introduction

The evaluation of NLP systems in Machine Translation and Automatic Summarization has reached a "local optimum." While $n$-gram overlap metrics like BLEU and ROUGE are widely criticized for failing to capture semantic nuances or complex syntax, they remain the industry standards. Proposing more sophisticated metrics—such as those based on dependency trees or semantic roles—is challenging because these novel measures often exhibit lower direct correlation with human scores on specific datasets compared to the established standards.

This research re-frames the problem: instead of seeking a single "perfect" metric, we should ask whether adding more metrics makes our overall conclusions more reliable.

Theoretical Framework: Reliability and Heterogeneity

The authors introduce two key properties to evaluate sets of metrics:

  1. Additive Reliability: This property posits that the reliability of a conclusion (e.g., "System A is better than System B") should increase, or at least not decrease, when corroborated by an additional similarity measure.
  2. Heterogeneity: This is defined as the probability that two metrics will contradict each other.

A core insight of the paper is that reliability is lower-bounded by heterogeneity. If two highly different (heterogeneous) metrics both agree on a system's superiority, that conclusion is statistically more robust than if two similar metrics (like BLEU and NIST) agree.

The positive correlation between the heterogeneity of metric pairs and the resulting reliability of the evaluation.

Methodology and Datasets

The study employs the ASIYA Toolkit to access a wide spectrum of metrics, ranging from surface-level lexical counts to deep syntactic and semantic features. Meta-evaluation was performed using established benchmarks:

  • Machine Translation: NIST MT Evaluation campaigns (2004-2005) for Arabic-to-English (AE) and Chinese-to-English (CE).
  • Automatic Summarization: Document Understanding Conference (DUC) 2005 and 2006 tasks.
  • Ground Truth: Human assessments of adequacy, fluency, and responsiveness were used to validate the reliability gains.

The Role of Linguistic Diversity

The experiments confirmed that linguistic metrics act as "force multipliers." While a metric focusing on semantic roles might have a lower standalone correlation with human scores, its high heterogeneity relative to lexical metrics makes it invaluable for corroboration.

Comparison of heterogeneity levels when combining different numbers of lexical measures versus combinations of linguistic and lexical measures.

As shown in the figure above, sets that include both lexical and linguistic metrics achieve significantly higher heterogeneity than sets restricted to lexical metrics alone. This diversity is the engine behind reliability gains.

Key Results and Discussion

The empirical results across both MT and AS tasks support the "universal reliability gain" hypothesis. In MT, the reliability gain reached up to 0.34 when using highly heterogeneous sets. In Summarization, gains were smaller (up to 0.08) but still positive, likely due to the higher redundancy among ROUGE variants.

Reliability vs. Sensitivity

The authors note a fundamental trade-off: increasing the number of heterogeneous metrics increases reliability (certainty of the result) but decreases sensitivity. It becomes harder for a system to show a "unanimous" improvement across all metrics simultaneously. However, the paper argues that requiring improvement across multiple metrics is a more effective way to ensure quality than simply demanding a larger point jump in a single, flawed metric.

Conclusion and Future Work

This work provides a theoretical and empirical basis for multi-metric evaluation. Future research could explore weighted combinations of heterogeneous signals or machine-learned meta-metrics to move beyond simple unanimity requirements. Despite the computational cost of deep linguistic metrics, their ability to provide robust corroboration makes them essential for the next generation of NLP evaluation.

Summary updated

BibTeX
@inproceedings{amigo2011corroborating,
  author = {Enrique Amigó and Julio Gonzalo and Jesús Giménez and Felisa Verdejo},
  title = {{Corroborating Text Evaluation Results with Heterogeneous Measures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
}
Figure