Abstract
In fields like Machine Translation (MT) and Automatic Summarization (AS), researchers often rely on a single metric, such as BLEU or ROUGE, despite their known limitations. This paper proposes a shift toward using a diverse "jury" of metrics. By formalizing the concepts of additive reliability and heterogeneity, the authors demonstrate that corroborating evaluation results with multiple, heterogeneous measures consistently increases the reliability of NLP evaluations. Even metrics with lower individual correlations to human judgments can significantly enhance the overall certainty of an evaluation conclusion if they capture different linguistic properties.
Introduction
The evaluation of NLP systems in Machine Translation and Automatic Summarization has reached a "local optimum." While $n$-gram overlap metrics like BLEU and ROUGE are widely criticized for failing to capture semantic nuances or complex syntax, they remain the industry standards. Proposing more sophisticated metrics—such as those based on dependency trees or semantic roles—is challenging because these novel measures often exhibit lower direct correlation with human scores on specific datasets compared to the established standards.
This research re-frames the problem: instead of seeking a single "perfect" metric, we should ask whether adding more metrics makes our overall conclusions more reliable.
Theoretical Framework: Reliability and Heterogeneity
The authors introduce two key properties to evaluate sets of metrics:
- Additive Reliability: This property posits that the reliability of a conclusion (e.g., "System A is better than System B") should increase, or at least not decrease, when corroborated by an additional similarity measure.
- Heterogeneity: This is defined as the probability that two metrics will contradict each other.
A core insight of the paper is that reliability is lower-bounded by heterogeneity. If two highly different (heterogeneous) metrics both agree on a system's superiority, that conclusion is statistically more robust than if two similar metrics (like BLEU and NIST) agree.
Methodology and Datasets
The study employs the ASIYA Toolkit to access a wide spectrum of metrics, ranging from surface-level lexical counts to deep syntactic and semantic features. Meta-evaluation was performed using established benchmarks:
- Machine Translation: NIST MT Evaluation campaigns (2004-2005) for Arabic-to-English (AE) and Chinese-to-English (CE).
- Automatic Summarization: Document Understanding Conference (DUC) 2005 and 2006 tasks.
- Ground Truth: Human assessments of adequacy, fluency, and responsiveness were used to validate the reliability gains.
The Role of Linguistic Diversity
The experiments confirmed that linguistic metrics act as "force multipliers." While a metric focusing on semantic roles might have a lower standalone correlation with human scores, its high heterogeneity relative to lexical metrics makes it invaluable for corroboration.
As shown in the figure above, sets that include both lexical and linguistic metrics achieve significantly higher heterogeneity than sets restricted to lexical metrics alone. This diversity is the engine behind reliability gains.
Key Results and Discussion
The empirical results across both MT and AS tasks support the "universal reliability gain" hypothesis. In MT, the reliability gain reached up to 0.34 when using highly heterogeneous sets. In Summarization, gains were smaller (up to 0.08) but still positive, likely due to the higher redundancy among ROUGE variants.
Reliability vs. Sensitivity
The authors note a fundamental trade-off: increasing the number of heterogeneous metrics increases reliability (certainty of the result) but decreases sensitivity. It becomes harder for a system to show a "unanimous" improvement across all metrics simultaneously. However, the paper argues that requiring improvement across multiple metrics is a more effective way to ensure quality than simply demanding a larger point jump in a single, flawed metric.
Conclusion and Future Work
This work provides a theoretical and empirical basis for multi-metric evaluation. Future research could explore weighted combinations of heterogeneous signals or machine-learned meta-metrics to move beyond simple unanimity requirements. Despite the computational cost of deep linguistic metrics, their ability to provide robust corroboration makes them essential for the next generation of NLP evaluation.
Summary updated