TL;DR
This paper presents unsupervised neural machine translation (NMT) systems for English-German and German-English translation tasks. They found that iterative back-translation outperforms online methods, cross-lingual embeddings are crucial for online training, and Byte Pair Encoding (BPE) vocabularies are superior to word-based ones for online NMT.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) has achieved state-of-the-art results, but typically requires large amounts of parallel data (sentences aligned in two languages). Acquiring such data is expensive and time-consuming. Unsupervised NMT aims to overcome this limitation by training translation models using only monolingual data (text in a single language). This paper tackles the challenge of building effective unsupervised NMT systems, specifically for the German-English language pair, by exploring and comparing various components of existing unsupervised NMT architectures. They aim to identify the most effective strategies for vocabulary, initialization, and training methods within the context of the WMT 2018 shared task.
Methods
The core of their approach is an iterative back-translation method, building upon recent advancements in unsupervised NMT. This involves:
- Shared Encoder-Decoder Architecture: A single NMT model is trained to translate in both directions (e.g., English to German and German to English) simultaneously, sharing parameters to leverage common linguistic structures.
- Iterative Back-Translation: This is the primary training strategy. The model is trained in iterations. In each iteration, the current model is used to generate "back-translations" of monolingual data. This synthetic parallel data is then used, along with the original monolingual data, to train the model for the next iteration. This process is initialized with a word-by-word translation model based on cross-lingual word vectors.
- Online Optimization: An alternative training strategy where back-translations are generated on-the-fly for each mini-batch using the current model parameters. This method is not initialized with a word-by-word translation model.
- Vocabulary Types: They investigate both word-level vocabularies and Byte Pair Encoding (BPE) sub-word units. BPE is a data-driven method for segmenting words into smaller units, which can help handle rare words and out-of-vocabulary (OOV) issues.
- Embedding Initialization: Crucially, they explore different ways to initialize the word embeddings:
- Random: Randomly initialized embeddings.
- Monolingual: Pre-trained embeddings from monolingual data.
- Cross-lingual: Pre-trained embeddings that capture relationships between languages (e.g., from MUSE).
- Frozen: Keeping pre-trained embeddings fixed during training.
- Embedding Augmentations: They experiment with two techniques to improve embedding integration:
- Gating Mechanism: A learned mechanism to interpolate between pre-trained and randomly initialized embeddings, allowing the model to adapt to task-specific information.
- Weight Normalization: A technique to normalize word embedding vectors, which can help stabilize training and prevent exploding gradients.
Datasets
- Training Corpora: Monolingual News Crawl articles from 2014 to 2017 were used for both German and English.
- 100 million sentences were used for pre-training word embeddings.
- 5 million sentences were used for translation model training.
- Evaluation Datasets:
- newstest2017
- newstest2018
Key Insights and Ideas
- Iterative Back-Translation is King: The iterative back-translation approach, initialized with a word-by-word model, consistently outperformed the online optimization method across various configurations. This suggests that a strong initial signal is vital for unsupervised NMT.
- Cross-lingual Embeddings are Essential (Especially for Online): For the online optimization method, using random or monolingual embeddings led to poor performance. Cross-lingual embeddings were crucial for achieving reasonable results, highlighting the importance of language-independent representations. Even for the batch method, cross-lingual embeddings provided a slight advantage.
- BPE Outperforms Words (for Online): For the online training strategy, BPE vocabularies (specifically 50k merge operations) yielded better BLEU scores than word-based vocabularies. This is likely due to BPE's ability to handle rare words and reduce the OOV rate.
- Fine-tuning Embeddings is Better than Freezing: Freezing pre-trained embeddings during training generally led to worse performance. The model learns task-specific information through the embedding component, suggesting that fine-tuning is beneficial.
- Gating and Weight Normalization Offer Limited Gains: While gating and weight normalization were explored as ways to improve embedding integration, they did not significantly outperform the baseline or, in some cases, even slightly degraded performance. This suggests that simply training the initialized embeddings is often sufficient.
Key Results
- The best performing unsupervised system achieved 18.1 BLEU on newstest2017 (German-English) and 15.4 BLEU on newstest2018 (German-English) using the iterative batch optimization method with a word-based vocabulary and cross-lingual embeddings.
- The online optimization method, when using BPE vocabularies (50k merges) and cross-lingual embeddings, improved over word-based vocabularies by up to 1.2% BLEU.
- The iterative batch method with a word-based vocabulary and cross-lingual embeddings achieved 14.9 BLEU on newstest2017 and 18.1 BLEU on newstest2018.
- The same configuration with online optimization and BPE (50k) achieved 13.1 BLEU on newstest2017 and 15.4 BLEU on newstest2018.
- The paper also reports results for longer training iterations, reaching 19.2% BLEU with the batch method after more updates.
Limitations and Future Work
- Data Scarcity: The authors note that the amount of data used was relatively small, and increasing the training data could lead to further improvements.
- Optimization Tuning: They suggest that more effort could be invested in tuning the optimization algorithm's hyperparameters for the online method.
- Adversarial Loss Ineffectiveness: The adversarial loss term, which worked well for LSTM-based models, did not integrate well with the Transformer architecture and did not yield improvements. This is attributed to the different output characteristics of Transformer encoders.
- Noise Model and Auto-Encoding: The necessity of noise models and auto-encoding losses in later stages of training was investigated and found to be not statistically significant, suggesting that focusing solely on the translation task after initial learning is viable.
- Future Work: The authors mention that sharing embeddings across languages and initializing them with monolingual word vectors (as noted by Lample et al., 2018) is an avenue for future research.
Summary updated