Introduction
Recent benchmarks in Machine Translation (MT), such as those from the annual Workshop on Machine Translation (WMT), have suggested that state-of-the-art AI systems have reached or even surpassed "human parity" for several language pairs. However, these conclusions are typically drawn from aggregated scores—averages taken over thousands of sentences. This paper argues that such averages can be deceptive, potentially masking significant individual failures. By shifting the focus to the segment level, the authors investigate whether human parity holds up when translations are evaluated sentence-by-sentence.
Methodology
The study employs a high-resolution version of Direct Assessment (DA). In standard DA, human evaluators rate translation adequacy on a scale of 0–100. To ensure reliability at the individual segment level, the authors collected a minimum of 15 independent human ratings per segment for a sample of approximately 540 segments per language pair. This approach was applied to the Facebook-FAIR system and professional human translators across three language pairs from the WMT19 News Translation Shared Task: English-Russian, English-German, and German-English.
Key Results and Analysis
The segment-level analysis reveals that "human parity" is often an illusion created by aggregate metrics. Even when a system achieves a high mean score, it may still fail on specific segments that humans find trivial.
Disjoint Challenges
The research identifies "disjointly" challenging segments—cases where the machine fails while the human succeeds, and vice versa. For example, in the English-Russian pair, the machine failed on 2.4% of segments that the human handled easily. Conversely, in the English-German and German-English pairs, there were instances where the machine significantly outperformed the human, often due to the human producing awkward "translationese" or making errors in tense and unit conversion.
Qualitative Error Profiles
Machine errors frequently involve syntactic deviations or a failure to grasp context-dependent polysemy (e.g., translating a weather "trough" as a feeding "Trog"). Human errors, on the other hand, are often related to stylistic awkwardness or subtle inaccuracies that arise from following the source text too closely. These findings suggest that human parity is not a point of convergence in capability, but rather a crossover point where different types of errors are being made.
Conclusion
This study demonstrates that aggregate scores are insufficient for assessing the true quality of machine translation. While AI systems are increasingly robust, they continue to stumble on specific linguistic phenomena that humans navigate effortlessly. Moving forward, segment-level evaluation should be adopted as a standard practice to identify and address the remaining gaps between machine and human translation performance.
Summary updated