TL;DR
GTCOM's submission for WMT19 news translation achieved top results by focusing on data augmentation and advanced decoding techniques. They leveraged back-translation, knowledge distillation, and a novel ensemble decoding strategy to significantly improve translation quality across multiple language pairs, particularly excelling in English-to-Gujarati and Lithuanian-to-English directions.
What Problem Does This Paper Address?
The paper tackles the challenge of building high-performing Neural Machine Translation (NMT) systems for the WMT19 shared news translation task. This involves translating text between various language pairs, often with limited parallel data, which is a common bottleneck in NMT. The goal is to push the state-of-the-art in machine translation quality, as measured by the BLEU score, by exploring effective data processing, model training, and decoding strategies. Specifically, they aim to improve translation for directions like English-Gujarati, English-Lithuanian, and English-Finnish, which can be considered low-resource compared to more common language pairs.
Methods
GTCOM's approach is a multi-pronged strategy centered around data enhancement and sophisticated decoding:
- Data Filtering: They employed a combination of human rules and language models to clean monolingual, parallel, and synthetic data. Sentences were scored by language models, and a weighted combination of source and target sentence scores was used to filter out low-quality data.
- Back-Translation: This standard technique was used to generate synthetic parallel data by translating monolingual data from the target language back into the source language. They trained target-to-source models and then used them to translate monolingual data.
- Knowledge Distillation (KD):
- Sequence-level KD: A smaller "student" model is trained to mimic the output of a larger "teacher" model. They explored different teacher models and applied this to improve translation quality.
- Sequence-level Interpolation: After KD, models were fine-tuned using "n-best" translations generated by the KD process.
- R2L Reranking: This technique involves using a reverse (target-to-source) model to rescore candidate translations from the primary (source-to-target) model. They improved upon previous attempts by systematically increasing the beam size and exploring combinations of R2L models.
- Ensemble Decoding: To achieve the best performance, they employed a GMSE (Generalized Minimum Bayes-Risk Estimation) algorithm to select and combine the outputs of multiple models, aiming for more robust and higher-quality translations.
Datasets
The paper utilizes data from the WMT19 shared task, which includes:
- Parallel Data: Europarl v9, ParaCrawl v3, Wiki Titles v1, Rapid corpus of EU press releases, Bible Corpus, OPUS, and custom crawled data.
- Monolingual Data: News crawl, Europarl, and Europarl v9.
- Synthetic Data: Generated through back-translation and knowledge distillation.
The newsdev2019 set was used as the development set.
Key Insights and Ideas
- Data is King (and Queen): The paper strongly emphasizes that augmenting and cleaning data, especially for low-resource languages, is crucial for NMT performance. Back-translation and knowledge distillation are powerful tools for this.
- R2L Reranking Needs Care: Simply applying R2L reranking isn't enough. The authors found that a systematic approach, involving increasing beam sizes and exploring model combinations, is key to unlocking its benefits.
- Ensembling for Robustness: Combining multiple models through ensemble decoding (specifically GMSE) proved to be a highly effective strategy for boosting translation quality beyond what any single model could achieve.
- Back-Translation's Power: Back-translation consistently yielded significant BLEU score improvements, especially in low-resource scenarios, highlighting its continued importance in NMT.
Key Results
GTCOM's systems achieved excellent results in WMT19, securing the best BLEU scores among participants for the English-to-Gujarati (28.2) and Lithuanian-to-English (36.3) directions.
Across various language pairs, the application of their techniques led to consistent improvements over a baseline model:
- Back-translation: Provided the most significant gains, ranging from 1.54 to 4.87 BLEU points.
- Joint Training: Offered smaller but still noticeable improvements (0.12 to 0.29 BLEU).
- Sequence-level KD: Improved BLEU scores by 0.09 to 1.03 points.
- Sequence-level Interpolation: Added a further 0.12 to 0.21 BLEU points.
- Ensemble Decoding: Achieved improvements of 0.22 to 0.55 BLEU points.
- R2L Reranking: Provided a final boost of 0.1 to 0.17 BLEU points.
For example, in English-to-Lithuanian, the final system with R2L reranking reached a BLEU score of 28.37, up from a baseline of 22.56.
Limitations and Future Work
- Time Constraints: The authors mention repeating the back-translation procedure only twice due to time limitations, suggesting further iterations might yield even better results.
- R2L Reranking Optimization: While improved, the paper implies that further exploration of R2L reranking strategies could still be beneficial.
- Data Augmentation Depth: The effectiveness of back-translation and KD suggests that exploring even more extensive data augmentation techniques could be a fruitful avenue for future research.
- Low-Resource Focus: While successful, the paper's focus on low-resource languages hints at ongoing challenges and opportunities in this area.
Summary updated