TL;DR
This paper presents a detailed linguistic evaluation of 16 German-English machine translation systems using a comprehensive test suite covering 107 grammatical phenomena. While overall performance has improved, systems still struggle with specific linguistic challenges like idioms and verb valency, highlighting the ongoing need for nuanced evaluation beyond simple accuracy scores.
What Problem Does This Paper Address?
For years, machine translation (MT) evaluation has largely relied on automatic metrics or broad human evaluations that provide a general quality score or ranking. While useful, these methods often fail to pinpoint why a translation is good or bad, especially for complex linguistic phenomena. Previous efforts to analyze translation quality in more detail have been unsystematic or limited to shallow error categories. Furthermore, many MT evaluations use generic text sources, assuming they represent typical translation tasks. This paper addresses the need for a more systematic and fine-grained approach to MT evaluation by employing a specialized test suite designed to probe specific linguistic phenomena. This allows for a deeper understanding of system strengths and weaknesses beyond overall accuracy.
Methods
The core of this research is the application of the DFKI test suite for German-English MT. This test suite is not a random sample of text but a carefully constructed set of sentences designed to trigger specific translation errors related to particular linguistic phenomena. The process involves several steps:
- Produce Paradigms: Linguists create or select source sentences that are likely to cause errors for specific phenomena.
- Fetch Sample Translations: These source sentences are translated by existing MT systems to gather initial outputs.
- Write Regular Expressions: Annotators analyze the MT outputs and develop positive and negative regular expressions to automatically identify correct (pass) or incorrect (fail) translations for each phenomenon.
- Fetch More Translations: The complete test suite is then applied to a larger set of MT systems, including those participating in a shared task.
- Apply Regular Expressions: The developed rules are automatically applied to the MT outputs to assess performance.
- Resolve Warnings and Refine Rules: Any ambiguities or errors in the automatic evaluation are manually resolved by annotators, and the rules are refined to improve robustness and coverage. This iterative process ensures a high degree of accuracy in the evaluation.
The paper also mentions the use of TQ-AutoTest software for processing the MT outputs and a one-tailed Z-test with $\alpha = 0.95$ for confirming the significance of comparisons between systems.
Datasets
The primary dataset used in this paper is the DFKI test suite for German-English MT. This test suite contains 5560 test sentences designed to evaluate 107 grammatical phenomena organized into 14 categories. The paper notes that this is the latest version of the test suite, previously presented in Burchardt et al. (2017) and applied in the previous year's shared task. The test items were created manually, distinguishing it from other test suites that might use automated methods for text extraction. The evaluation was performed on the outputs of 16 MT systems participating in the Fourth Conference on Machine Translation (WMT19) news translation task.
Key Insights and Ideas
- Beyond Overall Accuracy: The study strongly emphasizes that high overall accuracy doesn't guarantee good performance across all linguistic aspects. A significant portion of test items are still translated incorrectly, even by top-performing systems.
- Systematic Probing is Crucial: The test suite approach reveals specific weaknesses that broad metrics miss. For instance, while systems might perform well on average, they can fail spectacularly on specific phenomena like idioms.
- Idioms Remain a Major Hurdle: The results highlight a persistent and significant challenge in translating idioms, with very low accuracy scores for this category.
- Progress in Function Words and Agreement: The paper notes improvements in the translation of function words, non-verbal agreement, and punctuation compared to the previous year, indicating progress in these areas.
- Heterogeneity of Systems: The evaluation acknowledges the difficulty in drawing definitive conclusions about specific development settings due to the diverse nature of the participating MT systems (e.g., NMT vs. Transformer-based).
Key Results
- On average, the MT systems translated one out of four test items incorrectly, achieving an overall accuracy of 75.6%.
- Lowest performing categories included multi-word expressions (MWE) and verb valency, with around 66% accuracy.
- Idioms showed particularly poor performance, with systems translating only 11.6% of them correctly.
- Negation was a strong category, with accuracy above 80%, followed by composition, function words, subordination, and non-verbal agreement.
- Compared to the previous year, there was an overall improvement of about 6% in average accuracy.
- The largest improvements were observed in function words (+12.5%), non-verbal agreement (+9.7%), and punctuation (+8%).
- Facebook was identified as a strong performer, winning the most categories, while RWTH and Online-A led in average accuracy across all test items.
Limitations and Future Work
The authors acknowledge that interpreting the average accuracy scores requires caution, as they are derived from an artificial test suite rather than a naturalistic sample. The heterogeneity of the MT systems also limits the ability to directly attribute performance to specific development settings or design decisions.
For future work, the authors aim to:
- Investigate the impact of particular development settings and design decisions on specific phenomena.
- Conduct more in-depth comparative analyses of system settings, which was challenging for this shared task due to a lack of detailed system information.
Summary updated