TL;DR
The University of Edinburgh's WMT18 submission tackled all 14 language pairs in the news translation task, leveraging novel RNN variants, Transformer architectures, and advanced data selection/weighting techniques. They achieved strong results by combining these innovations, particularly in handling noisy parallel data and extending back-translation methods.
What Problem Does This Paper Address?
The core challenge addressed is improving neural machine translation (NMT) quality across a wide range of language pairs, specifically for the demanding WMT news translation task. This involves dealing with the inherent noise in large-scale parallel corpora, optimizing model architectures, and effectively utilizing available monolingual data through techniques like back-translation. The paper also implicitly addresses the challenge of scaling NMT systems to cover a large number of language pairs while maintaining high performance.
Methods
The University of Edinburgh's submission employed a multi-faceted approach, building upon established NMT techniques while introducing several novelties:
- Hybrid Architectures: They experimented with both the Transformer architecture (Vaswani et al., 2017) and a novel variant of deep RNNs, specifically a BiDeep GRU encoder-decoder. They also explored mixed ensembles of these architectures.
- Multi-Head and Multi-Hop Attention: For their RNN models, they extended the attention mechanism with multi-head and multi-hop capabilities, aiming to capture more complex dependencies in the source sentence.
- Data Selection and Weighting: Recognizing the noise in large parallel corpora, especially the newly introduced ParaCrawl data, they implemented several filtering strategies:
- Language Identifier Filtering: Using
langid.pyto remove sentence pairs where the language identification was inconsistent. - Translation Perplexity Filtering: Training shallow RNNs to score ParaCrawl sentences and filtering based on perplexity thresholds.
- Alignment-Based Filtering: Using
fastalignto compute alignment probabilities and filtering based on a "plausible translation" score. - Data Weighting: Applying different weights to various corpora during training, particularly for DE↔EN.
- Extensions to Back-Translation: They utilized copied monolingual data and iterative back-translation, especially for the TR↔EN pair, to generate synthetic parallel data.
- In-domain Fine-tuning: For RU↔EN, they constructed a specialized "in-domain" dataset by carefully selecting and filtering data to fine-tune their models, aiming to improve performance on the target domain.
- Preprocessing: They used the Moses pipeline for normalization, tokenization, and truecasing, followed by joint Byte-Pair Encoding (BPE). They also adapted preprocessing for non-Latin scripts.
- Training Techniques: They employed Adam optimizer, label smoothing, exponential smoothing, gradient clipping, layer normalization, and dropout.
Datasets
The primary datasets used were:
- Parallel Corpora: Standard WMT news translation task parallel data.
- ParaCrawl Corpus: A newly introduced noisy parallel corpus for many language pairs.
- Monolingual Corpora: News crawls from various years (e.g., 2016, 2017) were used for back-translation.
- Specific Corpora: NewsCommentary v13, UN data, CWMT, Europarl, CommonCrawl, Yandex Corpus, SETIMES2, and Rapid were utilized for specific language pairs or fine-tuning.
No new benchmarks were introduced; the focus was on effectively utilizing existing datasets.
Key Insights and Ideas
- Noise is a Major Hurdle: The introduction of the ParaCrawl corpus, while beneficial, highlighted the significant challenge posed by noisy parallel data. Effective filtering strategies were crucial for leveraging this data.
- Hybrid Models Offer Synergy: Combining RNNs and Transformers in ensembles showed promise, suggesting that different architectures capture different aspects of translation.
- Data Selection is Key: Sophisticated data selection and filtering techniques, beyond simple BPE, can significantly boost performance, especially when dealing with large, noisy datasets.
- In-domain Fine-tuning Pays Off: Creating and using specialized in-domain data for fine-tuning can lead to substantial improvements, as demonstrated for the RU↔EN pair.
- Back-translation is Still Powerful: Extending back-translation with copied monolingual data and iterative approaches proved effective, particularly for low-resource scenarios.
Key Results
- The University of Edinburgh submitted systems for all 14 language pairs in the WMT18 news translation task.
- They achieved strong performances across most pairs, often placing within the top constrained cluster.
- For DE↔EN, their single transformer model achieved a BLEU score of 44.4, outperforming their ensemble models.
- In-domain fine-tuning for RU↔EN showed a significant improvement, with BLEU scores increasing from 30.25 to 32.69 for deep RNNs and 35.81 to 36.96 for Transformers.
- Post-submission experiments indicated that multi-head/multi-hop attention provided a small positive effect on BLEU.
- Reducing vocabulary size had a mixed effect on RNNs but generally benefited Transformer models.
Limitations and Future Work
- Time Constraints: The authors acknowledge that due to time constraints, they were unable to train all models to convergence, which likely impacted performance on some language pairs.
- Transformer vs. RNN Performance: For DE↔EN, the single transformer model outperformed ensembles, suggesting that hyperparameter choices or ensemble strategies might not have been optimal.
- Vocabulary Size Impact: The exact relationship between BPE hyper-parameters and BLEU remains unclear, requiring further investigation.
- Future Work: The paper suggests further investigation into vocabulary size effects and the optimal mix of architectures in ensembles. The authors also note that covering all 14 language pairs limited their ability to deeply experiment on each.
Summary updated