TL;DR
This paper presents LMU Munich's systems for unsupervised machine translation (MT) between English and German, specifically for the WMT18 news translation task. They achieve competitive results by combining existing techniques like bilingual word embeddings, denoising, and back-translation, demonstrating the effectiveness of these methods even without parallel data.
What Problem Does This Paper Address?
Traditional machine translation systems heavily rely on vast amounts of parallel text (sentences and their translations). However, for many language pairs, this parallel data is scarce or non-existent, hindering the development of effective MT systems. This paper tackles the challenge of building high-quality MT systems when only monolingual data (text in a single language) is available, a scenario known as unsupervised machine translation. This is particularly relevant for low-resource languages and for pushing the boundaries of what's possible with limited data.
Methods
The LMU Munich team developed two main types of unsupervised MT systems:
- Phrase-Based Translation (PBT): This approach leverages bilingual word embeddings (BWEs) to create an initial phrase table. They also incorporated a target-side n-gram language model. A key innovation here was experimenting with translating German compound words by treating bigrams (two-word sequences) as single units, and vice-versa, to improve translation accuracy. They also explored using orthographic similarity between words to boost translation quality, especially for named entities.
- Neural Machine Translation (NMT): This system is based on recent advancements in unsupervised NMT, primarily drawing from the work of Lample et al. (2018b). The core idea is to train a standard NMT architecture using two main components:
- Denoising: The model learns to reconstruct clean sentences from noisy versions (e.g., with swapped words). This helps the model learn language fluency and also aids in handling word order differences between languages.
- On-the-fly Backtranslation: The model translates monolingual sentences from the target language back to the source language. This generated "synthetic" parallel data is then used to train the translation model. They also explored initializing NMT components with BPE-level embeddings and found that disabling the denoising component in later training stages could further improve performance.
Datasets
The primary datasets used were:
- NewsCrawl articles (2007-2017): This large corpus of news text in German and English was used for training. For experiments, they subsampled 4 million sentences per language. They also specifically used NewsCrawl 2017 for some experiments to assess the impact of more recent data.
- newstest2017 and newstest2018: These were used as test sets for evaluating the translation quality.
- Europarl and News Commentary: These datasets were excluded due to content overlap with NewsCrawl.
Key Insights and Ideas
- Bilingual Word Embeddings are Crucial: The foundation of both their phrase-based and neural systems relies heavily on well-trained bilingual word embeddings, which map words from different languages into a shared vector space.
- Orthographic Similarity Boosts Named Entity Translation: Incorporating orthographic similarity between words significantly improved the translation of named entities, a known challenge for BWE-based systems.
- Denoising and Backtranslation are Powerful NMT Tools: The combination of denoising and on-the-fly backtranslation is highly effective for training NMT models without parallel data, enabling them to learn language-specific nuances and word order.
- Training Curriculum Matters: The order and type of data used during training (e.g., starting with synthetic word-by-word translations, then focusing on backtranslation) can significantly impact performance and learning stability.
- Disabling Denoising Can Help: In later stages of NMT training, disabling the denoising component and focusing solely on backtranslation led to performance gains.
Key Results
- Their unsupervised NMT system achieved competitive results in the WMT18 unsupervised learning sub-track, outperforming their phrase-based system.
- Using orthographic similarity alone improved BLEU scores by up to 1.03 for de-en and 0.30 for en-de in word-by-word translations.
- Combining orthographic similarity and bigrams yielded the best results for German→English translation, improving upon orthographic similarity alone by 0.92 BLEU.
- Disabling denoising in the NMT system provided further gains of 1.26 BLEU for de-en and 1.63 BLEU for en-de.
- Their best unsupervised NMT system achieved 13.77 BLEU for de-en and 10.45 BLEU for en-de on newstest2017.
Limitations and Future Work
- Named Entity Translation Challenges: The paper acknowledges that translating named entities remains a challenge, even with orthographic similarity, due to potential confusion between similar-looking entities.
- Impact of Data Recency: While using more recent data (NewsCrawl 2017) showed benefits, the paper couldn't definitively pinpoint whether this was due to better content overlap or the recency itself.
- Unstable Learning: They observed initial training difficulties with their NMT model, suggesting that careful control of the training procedure is necessary.
- Future Work: The authors suggest that further exploration of training curricula and data selection strategies could lead to even better performance. The analysis of example translations also highlights areas for future improvement, such as handling more complex reordering and idiomatic expressions.
Summary updated