TL;DR
The Air Force Research Laboratory (AFRL) team improved their Russian-English machine translation systems for WMT18 by exploring various neural network architectures, training strategies, and system combination techniques. They found that using a specialized validation metric (BEER) and pretrained word embeddings significantly boosted performance, leading to a strong final submission through ensembling.
What Problem Does This Paper Address?
Building high-quality machine translation (MT) systems is a complex task that involves navigating a vast landscape of architectural choices, training data, and optimization strategies. For the WMT news translation shared task, specifically the Russian-English language pair, the challenge is to leverage recent advancements in neural MT while adapting them to the specific domain and data available. The AFRL team aimed to improve their existing MT systems by systematically investigating these factors, building upon their previous year's work and incorporating new developments in the field. This includes exploring different neural network architectures like "bi-deep" and Transformer models, optimizing training procedures, and effectively combining multiple systems to achieve the best possible translation quality.
Methods
The AFRL team focused on three primary MT toolkits: Marian, OpenNMT, and Moses.
- Marian: This was the primary focus for experimentation. They explored variations in:
- Validation Metrics: Comparing cross-entropy, BLEU, and BEER (a trained MT evaluation metric) for validation.
- Pretrained Word Embeddings: Investigating the impact of using word2vec embeddings trained on large monolingual corpora.
- Training Corpora: Experimenting with different combinations of WMT17 and WMT18 data, including back-translated data.
- Fine-tuning: Applying continued training on a curated corpus of past WMT test sets.
- Architectures: Testing both "bi-deep" and Transformer models with consistent settings for fair comparison.
- Constant Settings: To isolate the effects of specific variables, they kept joint BPE splits (49500), vocabulary size (90k), word embedding dimensionality (512), hidden layer size (1024), and validation set (newstest2014) constant for many experiments.
- OpenNMT: This system used a standard RNN architecture and was fine-tuned on available news task test sets.
- Moses: A phrase-based MT system was trained, incorporating a hierarchical reordering model and a 5-gram operation sequence model.
- System Combination: The "Jane" system combination method was used to merge the outputs of the individual best systems (Marian, OpenNMT, and Moses) to create the final submission.
Datasets
- Training Data:
- WMT17 parallel corpus.
- WMT18 preprocessed data provided by organizers.
- Paracrawl corpus (Russian-English portion, noted as noisy).
- Monolingual CommonCrawl data (Russian and English) for training word embeddings.
- A fine-tuning corpus constructed from WMT news task test sets from 2013-2017.
- Validation Data: newstest2014 was consistently used for validation during Marian system development.
- Test Data: Various newstest sets (newstest2013 through newstest2018) were used for evaluation.
Key Insights and Ideas
- BEER as a Validation Metric is Crucial: Using BEER as a validation metric, rather than traditional cross-entropy or BLEU, yielded significant improvements in BLEU scores (up to +1.5 BLEU). This highlights the importance of using evaluation metrics that better correlate with human judgment during the training process.
- Pretrained Embeddings Provide a Strong Boost: Incorporating pretrained word embeddings (trained with word2vec) resulted in a substantial BLEU improvement (over +1.0 BLEU) without altering other training parameters. This suggests that leveraging external linguistic knowledge is highly beneficial.
- Corpus Choice Matters, But Not Always Simply More is Better: While concatenating WMT17 and WMT18 data generally improved performance, the exact combination and its interaction with other factors (like validation metrics) required careful tuning.
- Fine-tuning Offers Further Gains: A brief examination of fine-tuning on a corpus of past test sets showed a promising increase in BLEU (+3), though concerns about overfitting were noted.
- Ensembling is Key for Top Performance: The final submission, a combination of multiple well-tuned systems, achieved the highest scores, demonstrating the continued effectiveness of system combination in MT.
Key Results
- Using BEER as a validation metric for Marian systems resulted in an increase of +0.7 to +1.5 BLEU compared to cross-entropy or BLEU.
- Pretrained word embeddings led to an improvement of over +1.0 BLEU across available test sets.
- Combining WMT17 and WMT18 training data showed an increase of +0.7 to +1.5 BLEU for test sets not used for validation.
- Fine-tuning on a corpus of past test sets resulted in a gain of almost +3 BLEU on newstest2018.
- The final system combination (Syscomb) achieved a BLEU score of 30.01 and a BEER score of 0.597 on newstest2018.
Limitations and Future Work
- Overfitting Concerns: The authors noted potential overfitting to the fine-tuning corpus, suggesting that more careful regularization or cross-validation might be needed for this technique.
- Limited Exploration of Transformer Architectures: While Transformer models were used, the majority of the experimental effort was focused on the "bi-deep" architecture. Further exploration of Transformer variants could yield additional improvements.
- Domain Specificity: The focus was on the Russian-English news translation task. Adapting these findings to other language pairs or domains might require different strategies.
- Future Work: The paper implicitly suggests continued exploration of fine-tuning techniques, further architectural experimentation, and potentially more sophisticated methods for combining diverse systems.
Summary updated