TL;DR
The WMT 2019 Automatic Post-Editing (APE) shared task explored systems that automatically correct machine translation (MT) outputs. This year, the focus was on neural MT outputs for English-German and English-Russian, with a domain focus on Information Technology. While English-German saw significant progress, the English-Russian task proved challenging due to the high quality of the initial MT, highlighting the difficulty of improving already good translations.
What Problem Does This Paper Address?
Machine translation (MT) systems have become remarkably good, but they still produce errors. Automatic Post-Editing (APE) aims to automatically correct these errors, reducing the effort required by human translators. This is crucial for making MT practical in real-world applications, especially when dealing with specialized domains or when the MT system's internal workings are inaccessible. The challenge lies in developing APE systems that can effectively fix errors without introducing new ones or unnecessarily altering already correct translations, particularly as MT quality continues to improve.
Methods
The WMT 2019 APE task followed a consistent evaluation setup from previous years. Participants were tasked with building systems that learn from human corrections to automatically edit MT outputs. The core methodology involved training models on datasets containing source sentences, their MT translations, and corresponding human post-edits.
Key methodological trends observed among participants included:
- Neural Architectures: The vast majority of systems employed neural network architectures, with the Transformer model (Vaswani et al., 2017) being the dominant choice, often used directly or adapted.
- Multi-Source Approach: Many systems leveraged a multi-source approach, utilizing information from both the original source sentence and the MT output to be corrected.
- Synthetic Data: The use of synthetic data, either provided as part of the task or generated by participants, was common for augmenting training data.
- Over-Correction Mitigation: A significant focus for some teams was on preventing "over-correction," where systems might unnecessarily modify acceptable translations. Techniques included incorporating special tokens to signal the expected amount of post-editing or using penalties during decoding.
Datasets
The task utilized data from the Information Technology (IT) domain for two language pairs:
- English-German: This subtask reused the test set from the 2018 round, allowing for direct comparison of progress. Training and development sets contained 13,442 and 1,000 triplets, respectively. Additional resources included the "Artificial" corpus (4.5 million artificially-generated post-editing triplets) and the eSCAPE corpus (14.5 million instances).
- English-Russian: This was a new subtask. Training and development sets contained 15,089 and 1,000 triplets, respectively, with a test set of 1,023 instances. The data was sourced from Microsoft Office localization data. The eSCAPE corpus was also extended to provide additional training material.
Key Insights and Ideas
- Progress in English-German APE: The English-German subtask demonstrated significant advancements, with four teams outperforming last year's winning system. This suggests that APE technology is maturing for this language pair.
- The "High-Quality MT" Challenge: The English-Russian subtask highlighted a critical challenge: improving already high-quality MT outputs is considerably harder. Even the best systems struggled to beat the baseline, indicating that APE systems need to be more precise and conservative when dealing with near-perfect translations.
- Repetitiveness vs. Quality: While high text repetitiveness in data was previously thought to aid APE systems, this year's results suggest its influence is marginal, especially when dealing with high-quality MT. The quality of the initial MT output appears to be a more dominant factor in task difficulty.
- Over-Correction Remains a Problem: Mitigating the tendency of APE systems to over-correct acceptable translations is still a key open problem. This is particularly relevant as MT systems improve.
- Human Evaluation is Key: While automatic metrics are useful, human evaluation (using direct assessment) remains crucial for a nuanced understanding of system performance, especially for assessing subtle quality differences and the impact of over-correction.
Key Results
- English-German:
- Seven teams participated, submitting 18 runs.
- Four teams achieved better results than the 2018 winning system.
- The top-ranked system achieved -0.78 TER and +1.23 BLEU points improvement over the baseline.
- Improvements for the top four primary submissions were not statistically significant, indicating widespread progress.
- English-Russian:
- Two teams participated, submitting 2 runs each.
- None of the submitted runs beat the baseline (16.16 TER, 76.20 BLEU).
- The top submission achieved 16.59 TER and 75.27 BLEU, with differences from the baseline being statistically significant.
- This subtask was considered more challenging due to the high quality of the initial MT.
- Human Evaluation (English-German):
- Human post-editing outperformed all automatic post-editing systems.
- UNBABEL achieved the best APE performance according to human evaluators.
- All but one APE system outperformed unedited NMT output.
Limitations and Future Work
- English-Russian Difficulty: The high quality of the NMT output for English-Russian made the task significantly more challenging. Future work should focus on developing more precise and conservative APE solutions that can handle near-perfect translations and avoid unnecessary edits.
- Over-Correction: The tendency of systems to over-correct acceptable MT output remains a challenge. Future research should explore methods to better distinguish between necessary corrections and unnecessary modifications, potentially through more sophisticated modeling of post-editing style or quality prediction.
- Data Sparsity for Morphology-Rich Languages: The English-Russian subtask's difficulty might be exacerbated by the morphology-rich nature of Russian, potentially leading to sparser training data compared to English-German.
- Limited Human Evaluation for English-Russian: Due to low participation, no human evaluation was conducted for the English-Russian subtask. Future efforts could include this to gain a more complete understanding of system performance.
Summary updated