Abstract
Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.
Introduction: The "Tie" Problem in MT Meta-Evaluation
In Machine Translation (MT), automated metrics are essential for evaluating translation quality. To ensure these metrics are reliable, we perform meta-evaluation by measuring their correlation with expert human judgments. Historically, the field has relied on Kendall’s $\tau$, a statistic that evaluates pairwise rankings: if a human prefers translation A over B, does the metric agree?
However, as MT systems improve, they increasingly produce translations of similar quality, leading to a high frequency of "ties" in human judgments. In modern datasets like WMT’22, over 50% of translation pairs are often ties. Existing variants of Kendall’s $\tau$ handle these ties inconsistently—some ignore them, while others penalize them. This inconsistency allows for "gaming" the evaluation; for example, a metric can artificially inflate its correlation by predicting ties for difficult cases, effectively removing them from the calculation.
Proposed Methodology
To address these issues, this work proposes two fundamental changes to metric meta-evaluation:
1. Pairwise Accuracy ($acc_{eq}$)
Instead of complex correlation coefficients, the authors propose treating meta-evaluation as a three-way classification task. For any pair of translations, a metric must predict one of three states: $A > B$, $B > A$, or $A = B$. Pairwise accuracy ($acc_{eq}$) is the proportion of pairs where the metric’s prediction matches the human judgment, explicitly rewarding the correct identification of ties. This approach is more interpretable than $\tau$, representing the simple percentage of correct pairwise decisions.
2. Tie Calibration
Most neural metrics (e.g., COMET, BLEURT) produce continuous scores, making exact ties ($A = B$) statistically rare. To bridge the gap between continuous metrics and discrete human judgments, the authors introduce a tie threshold, $\epsilon$. If the absolute difference between two metric scores is less than $\epsilon$, the pair is treated as a tie.
The authors provide an efficient $O(n^2 \log n)$ algorithm to find the "optimal" $\epsilon$ that maximizes a metric's accuracy on a given dataset. This calibration allows for a fair comparison between "continuous" regression-based metrics and "discrete" metrics (like LLM-based prompting or span-based models).
Experimental Setup and Results
The study utilizes data from the WMT’22 Metrics Shared Task, focusing on high-quality Multidimensional Quality Metrics (MQM) ratings across language pairs such as English-German (en-de) and Chinese-English (zh-en).
Key Findings
- Leaderboard Shifts: Transitioning to calibrated $acc_{eq}$ significantly alters metric rankings. Metrics that naturally produce many ties, such as LLM-based GEMBA, were previously penalized by traditional $\tau$ variants. The new method provides a more level playing field.
- Fairer Comparisons: Once calibrated, continuous metrics often outperform discrete ones, demonstrating that they were always capable of identifying ties but required a threshold to express them.
- Robustness: $acc_{eq}$ proves more consistent across different data grouping strategies, leading to more stable and reliable meta-evaluations.
Conclusion and Limitations
By treating ties as meaningful signal rather than noise, $acc_{eq}$ and tie calibration provide a more robust framework for MT metric evaluation. While the optimal threshold $\epsilon$ may vary across datasets, treating it as a latent variable to be calibrated per test set offers a path forward. Future work may explore relative thresholds and address the transitivity of pairwise tie decisions.
Summary updated