Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

Daniel Deutsch, George F. Foster, Markus Freitag

Abstract

Kendall’s tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

Introduction

Machine Translation (MT) evaluation relies on automatic metrics to provide scalable quality assessments. To validate these metrics, researchers perform "meta-evaluations," typically measuring the correlation between metric scores and human judgments. Kendall’s $\tau$ is a standard choice for this task due to its intuitive focus on pairwise rankings. However, as MT systems reach human parity, the prevalence of "ties"—cases where two translations are judged of equal quality—presents a significant challenge to traditional correlation measures.

Figure 1: Comparison of two metrics against human scores. Metric 2 achieves perfect rank correlation ($\tau_b=1.00$) despite its coarse nature, illustrating how standard measures can be sensitive to tie handling and potentially misleading.

The Limitations of Kendall’s $\tau$

Standard correlation measures handle ties inconsistently, leading to three primary issues identified by the authors:

  1. Inconsistency: Different variants of $\tau$ (e.g., $\tau_b$, $\tau_{10}$) treat ties in contradictory ways, which can lead to conflicting rankings of metrics in shared tasks.
  2. The NaN Problem: Some variants become undefined (NaN) if a metric predicts ties for all segments in a set. This creates a loophole where researchers might "game" the system by forcing ties on difficult examples to artificially boost reported performance.
  3. Bias against Honesty: Metrics that correctly identify a human tie are often penalized by statistics that only reward correct directional rankings, treating a predicted tie as a "wrong" answer.

Proposed Methodology

To address these flaws, the paper introduces a more robust meta-evaluation framework centered on pairwise accuracy and calibration.

Pairwise Accuracy ($acc_{eq}$)

The authors advocate for a simple, intuitive accuracy score. It measures the proportion of pairs where the metric correctly predicts the human ranking or correctly predicts a tie. This measure is robust, never results in NaNs, and prevents the gaming of benchmarks by ensuring every pair is accounted for.

Tie Calibration

Modern regression-based metrics like COMET produce continuous floating-point scores, making exact ties extremely rare. To allow for fair comparison with discrete metrics (like LLM-based evaluators), the authors introduce a calibration procedure. This method determines an optimal threshold, $\epsilon$, for each metric. If the absolute difference between two scores is less than $\epsilon$, they are treated as a tie.

Figure 6: Tie calibration process. The optimal epsilon is selected by balancing accuracy and F1 scores for tie prediction, ensuring metrics are evaluated at their most effective operating point.

Experimental Evaluation

The framework was tested using data from the WMT’22 Metrics Shared Task, encompassing expert Multidimensional Quality Metrics (MQM) ratings across several language pairs (English-German, Chinese-English, English-Russian). The study evaluated a wide array of modern metrics, including Metric-X, COMET-22, BLEURT-20, and GPT-based evaluators like GEMBA.

Results and Discussion

The analysis reveals that ties are increasingly common, often exceeding 50% of pairs in high-quality datasets. Implementing calibrated pairwise accuracy ($acc_{eq}^*$) leads to a significant reshuffling of metric leaderboards compared to old standards like $\tau_{10}$.

Key findings include:

  • Fairness for LLMs: Metrics like MaTESe and GEMBA, which naturally produce many ties, were previously being unfairly penalized. Under the new framework, they are shown to be much more competitive.
  • Metric Interpretability: The calibrated $\epsilon$ provides a meaningful "minimum significant difference" for each metric, defining the point at which a score change actually reflects a likely change in human-perceived quality.
  • Robustness: Pairwise accuracy effectively closes the "NaN gaming" loophole, providing a more stable assessment of metric performance.

Conclusion

By properly accounting for ties through pairwise accuracy and calibration, the MT community can achieve fairer and more transparent meta-evaluations. This approach discourages benchmark gaming and provides a clearer picture of which metrics truly align with human judgment as translation quality continues to improve.

Summary updated

Abstract

Kendall’s tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.
BibTeX
@inproceedings{deutsch2023ties,
  author = {Daniel Deutsch and George F. Foster and Markus Freitag},
  title = {{Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=uDfPDlw8BG},
}
Figure