Abstract
Standard evaluation of Machine Translation (MT) metrics often relies on flawed statistical assumptions, leading to overstated performance claims. This paper identifies two primary issues: the use of symmetric confidence intervals for skewed correlation coefficients and the "small N" problem inherent in limited benchmark datasets. We propose a two-pronged solution: the application of rigorous statistical tests (Fisher r-to-z and Zou’s method) and the introduction of hybrid super-sampling. The latter generates thousands of synthetic MT systems from existing data, enabling a high-resolution, conclusive ranking of metric performance without additional human labeling.
Introduction
Automatic metrics such as BLEU, METEOR, and TER are essential for the iterative development of Machine Translation systems. To validate these metrics, researchers typically measure their correlation with human judgments across a set of MT systems. However, current evaluation practices suffer from two major hurdles. First, "statistical sloppiness" leads to the use of symmetric confidence intervals for correlation coefficients, which are mathematically skewed as they approach 1.0. Second, the "Small $N$" problem—where benchmarks contain only a few systems (e.g., 5 to 22)—makes it statistically difficult to distinguish between metric performances, resulting in inconclusive rankings where many metrics appear tied.
Methodology
Rigorous Statistical Testing
To address the skewed nature of correlation distributions, we advocate for the Fisher r-to-z transformation. Standard bootstrapping often produces symmetric intervals that can lead to impossible results, such as correlation upper bounds exceeding 1.0. Furthermore, when comparing two metrics evaluated on the same data, the correlations are "dependent." We implement Zou’s (2007) method to correctly calculate confidence intervals for the difference between these dependent correlations, ensuring more accurate significance testing.
Hybrid Super-Sampling
To overcome small sample sizes, we introduce hybrid super-sampling. By treating MT system outputs as a pool of segments, we can create "hybrid" systems by randomly selecting segments from different existing systems. With $n$ systems and $m$ sentences, the potential for synthetic systems is vast. We sample 10,000 such hybrids. Since human adequacy scores are collected at the segment level, we can compute a "true" human score for each hybrid, providing a massive dataset for metric validation without additional human effort.
Results and Analysis
Correcting Statistical Over-optimism
Applying rigorous testing to the WMT-15 German-to-English task revealed significant discrepancies in reported metric superiority. We found that 91 differences between metrics previously deemed "significant" under standard interpretations were actually not statistically significant when using proper dependent correlation tests. This highlights how standard methods frequently overstate the performance gaps between metrics.
Establishing Conclusive Rankings
In the WMT-12 Spanish-to-English task, the original evaluation was largely inconclusive due to the small number of systems. By applying hybrid super-sampling to generate 10,000 systems, we established a near-total ordering of metrics. This process identified TerrorCat as the clear winner, maintaining a tight, linear correlation with human scores even as other metrics like BLEU became noisy and failed to distinguish system quality effectively at certain performance levels.
Discussion and Conclusion
Our findings highlight that the rank order of metrics in small-sample evaluations is often fragile and noisy. By avoiding the "symmetry trap" and leveraging data remixing through hybrid super-sampling, researchers can achieve much higher evaluation power. This framework not only provides more reliable rankings but also offers a high-resolution environment for optimizing future MT metrics to better align with human judgment.
Summary updated