Introduction
Recent claims that machine translation (MT) has reached "human parity"—particularly in Chinese-to-English news translation—have sparked significant debate. This paper argues that these claims are often the result of flawed evaluation methodologies. By identifying two critical issues—the distorting effect of "translationese" and a lack of statistical power—the authors demonstrate that human translators still maintain a significant lead over the best MT systems.
The Problem of Translationese
A major flaw in standard MT evaluation is the use of "reverse-created" test data. This occurs when an English-to-Chinese system is evaluated using English text that was originally translated from Chinese. This resulting text, known as translationese, tends to be simpler, more explicit, and more regular than natural language, making it an artificially easy target for machines.
As shown in Figure 1, human judges consistently award higher scores to translations when the source is translationese (the "Reverse" condition). This creates an upward bias in perceived MT quality, acting as a "cheat code" that makes systems appear more capable than they are on natural, "forward-created" text.
The "Absence of Evidence" Fallacy
The second major issue is statistical power. Many studies claim "parity" simply because a statistical test failed to find a significant difference between a human and a machine. However, this is often a Type II error: the study lacked a large enough sample size to detect the human's superiority.
To address this, the authors recommend a statistical power standard of 0.8, a common benchmark in the behavioral sciences. For segment-level evaluations, they suggest a minimum of 385 distinct translations to avoid false parity claims caused by underpowered tests.
Re-evaluating Human Parity
The authors performed a rigorous re-evaluation of the 2018 Chinese-to-English systems that originally claimed human parity. By using only forward-created data, providing judges with full document context, and ensuring a large sample size ($n \approx 900$ distinct segments), they found that the human reference (REF-HT) significantly outperformed the top MT system (80.3 vs. 76.5, $p < 0.05$). This result confirms that when evaluations are sufficiently powered and use natural source text, the gap between humans and machines remains clear.
Reliability of Metrics
The study also highlights the unreliability of automatic metrics like BLEU, especially when dealing with translationese.
As illustrated in Figure 4, human Direct Assessment (DA) scores show much higher and more stable correlations across different statistical measures compared to BLEU. This reinforces the paper's argument that high-powered human evaluation is the only reliable way to assess true quality at the human-machine boundary.
Conclusion
The findings of this paper have already influenced the field, leading the Conference on Machine Translation (WMT) to officially cease using reverse-created test data in 2019. While achieving high statistical power and providing document context is more resource-intensive, it is essential for making valid claims about the progress of machine translation toward true human-level performance.
Summary updated