Statistical Power and Translationese in Machine Translation Evaluation

Yvette Graham, Barry Haddow, Philipp KoehnEMNLP 2020 · OpenReview

Introduction

Recent claims that machine translation (MT) has reached "human parity"—particularly in Chinese-to-English news translation—have sparked significant debate. This paper argues that these claims are often the result of flawed evaluation methodologies. By identifying two critical issues—the distorting effect of "translationese" and a lack of statistical power—the authors demonstrate that human translators still maintain a significant lead over the best MT systems.

The Problem of Translationese

A major flaw in standard MT evaluation is the use of "reverse-created" test data. This occurs when an English-to-Chinese system is evaluated using English text that was originally translated from Chinese. This resulting text, known as translationese, tends to be simpler, more explicit, and more regular than natural language, making it an artificially easy target for machines.

Figure 1: Human evaluation scores are consistently higher on reverse-created data (translationese) compared to original source text across various language pairs.

As shown in Figure 1, human judges consistently award higher scores to translations when the source is translationese (the "Reverse" condition). This creates an upward bias in perceived MT quality, acting as a "cheat code" that makes systems appear more capable than they are on natural, "forward-created" text.

The "Absence of Evidence" Fallacy

The second major issue is statistical power. Many studies claim "parity" simply because a statistical test failed to find a significant difference between a human and a machine. However, this is often a Type II error: the study lacked a large enough sample size to detect the human's superiority.

To address this, the authors recommend a statistical power standard of 0.8, a common benchmark in the behavioral sciences. For segment-level evaluations, they suggest a minimum of 385 distinct translations to avoid false parity claims caused by underpowered tests.

Re-evaluating Human Parity

The authors performed a rigorous re-evaluation of the 2018 Chinese-to-English systems that originally claimed human parity. By using only forward-created data, providing judges with full document context, and ensuring a large sample size ($n \approx 900$ distinct segments), they found that the human reference (REF-HT) significantly outperformed the top MT system (80.3 vs. 76.5, $p < 0.05$). This result confirms that when evaluations are sufficiently powered and use natural source text, the gap between humans and machines remains clear.

Reliability of Metrics

The study also highlights the unreliability of automatic metrics like BLEU, especially when dealing with translationese.

Figure 4: Comparison of correlation coefficients (Pearson, Spearman, Kendall) for BLEU and Human scores, showing that human judgments are more consistent and reliable.

As illustrated in Figure 4, human Direct Assessment (DA) scores show much higher and more stable correlations across different statistical measures compared to BLEU. This reinforces the paper's argument that high-powered human evaluation is the only reliable way to assess true quality at the human-machine boundary.

Conclusion

The findings of this paper have already influenced the field, leading the Conference on Machine Translation (WMT) to officially cease using reverse-created test data in 2019. While achieving high statistical power and providing document context is more resource-intensive, it is essential for making valid claims about the progress of machine translation toward true human-level performance.

Summary updated

BibTeX
@inproceedings{graham2020statistical,
  author = {Yvette Graham and Barry Haddow and Philipp Koehn},
  title = {{Statistical Power and Translationese in Machine Translation Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=SQZesyfYLbM},
}
Figure