TL;DR
The Air Force Research Laboratory (AFRL) team improved their machine translation systems for the WMT19 Russian-English task by refining data preprocessing, particularly with language ID filtering, and exploring subword segmentation strategies. They demonstrated that combining established techniques with careful tuning and system combination can yield competitive results.
What Problem Does This Paper Address?
Machine translation (MT) systems, especially for low-resource language pairs like Russian-English, face challenges with data quality and vocabulary limitations. The WMT shared task provides a benchmark for evaluating these systems. AFRL aimed to improve their MT performance by addressing these issues, building upon their previous year's work. This includes handling the "closed-vocabulary problem" where words not seen during training can't be translated, and optimizing how text is broken down into smaller units (subword segmentation) to better represent rare words and improve translation quality.
Methods
AFRL employed a multi-pronged approach:
- Data Preprocessing: A key innovation was using a pre-built language ID model from fastText to filter training data. Pairs where either the source or target sentence had a low probability of belonging to the intended language were discarded. This significantly reduced noisy data, especially in large, uncurated corpora like Paracrawl.
- Subword Segmentation: They explored the impact of different Byte-Pair Encoding (BPE) merge operation counts and vocabulary sizes. For their Marian systems, they found that 40k BPE merge operations yielded the best results.
- Domain Adaptation: For the Sockeye system, they investigated "continued training" to adapt a model to a new domain (e.g., patent applications) and mitigated the resulting performance drop on the original domain using Elastic Weight Consolidation (EWC).
- System Combination: They leveraged the Jane system combination method to merge outputs from their best individual MT systems (Marian, Sockeye, OpenNMT, Moses), aiming to achieve better overall performance than any single system.
- Ensemble Decoding: For Marian, they used an ensemble of multiple trained models, further enhanced by tuning feature weights using the "Expected Corpus BLEU" (ECB) metric.
Datasets
- Training Data:
- Common Crawl
- newscommentary
- Yandex
- Paracrawl
- UN2016
- WMT17 Russian-English system data (for OpenNMT-T)
- Monolingual English news-crawl data (for Moses language model)
- Test Sets: newstest2014, newstest2015, newstest2016, newstest2017, newstest2018, and newstest2019.
- Validation/Tuning Sets: newstest2014 and newstest2017 were used for tuning Marian systems. newstest2014 was used for validation in Marian training.
Key Insights and Ideas
- Language ID Filtering is Crucial: Aggressively filtering training data based on language identification significantly improves translation quality, especially for noisy, large-scale corpora. This is a practical and effective way to clean data.
- Subword Granularity Matters: The choice of BPE merge operations and vocabulary size has a tangible impact on performance, and empirical tuning is necessary to find the optimal setting for a specific language pair.
- Domain Adaptation Requires Care: While continued training can adapt models to new domains, it can lead to overfitting. Techniques like EWC are valuable for balancing adaptation with generalization.
- Ensembling and Combination Pay Off: Combining multiple well-trained systems, especially with tuned weights, consistently outperforms individual models. This highlights the robustness gained from diverse approaches.
Key Results
- Language ID filtering on the Paracrawl corpus removed 57.1% of lines and led to an approximate +1 BLEU point improvement for baseline Marian systems.
- The optimal BPE merge operation count for Russian-English was found to be 40k.
- Tuning Marian ensemble weights using ECB improved performance by up to +0.9 BLEU over a non-tuned ensemble.
- The best Sockeye EWC system showed improved performance on the out-of-domain newstest2018 compared to a continued-training system.
- The final 5-system combination achieved a BLEU score of 32.12 on newstest2018 and 37.2 on newstest2019.
Limitations and Future Work
- The authors note that the choice of the 0.8 probability threshold for language ID filtering was arbitrary and suggest future work to explore varying this threshold.
- The paper mentions that using newstest2017 as a tuning set for Marian did not help performance, while newstest2014 did, suggesting that the choice of tuning set can influence results.
- Future work could explore more advanced domain adaptation techniques and further investigate the impact of different subword segmentation strategies.
Summary updated