TL;DR
This paper introduces Cross-model Back-translated Distillation (CBD), a novel technique to enhance unsupervised machine translation (UMT) by generating more diverse synthetic data. CBD achieves state-of-the-art results on several UMT benchmarks, demonstrating its effectiveness in pushing the boundaries of translation quality without parallel data.
What Problem Does This Paper Address?
Machine translation (MT) systems typically require vast amounts of parallel data (sentences and their translations). However, acquiring such data is expensive and time-consuming. Unsupervised Machine Translation (UMT) aims to overcome this by learning to translate using only monolingual data. Existing UMT methods rely on principles like model initialization, language modeling (denoising auto-encoding), and iterative back-translation (IBT) to generate synthetic parallel data. While these methods have made significant progress, the authors observe that the performance gains from these diversification processes seem to have plateaued. They propose CBD to introduce a new level of data diversification that existing methods lack, aiming to further improve UMT performance.
Methods
The core innovation is Cross-model Back-translated Distillation (CBD). It builds upon existing UMT frameworks by introducing a fourth principle. Here's how it works:
- Two UMT Agents: Two independent UMT models (agents) are trained using standard UMT approaches. These agents are initialized with different parameters.
- Cross-Model Translation: One agent translates monolingual data from language A to B. Then, the other agent translates this generated data back from B to A. This "cross-model" aspect is crucial, meaning the agent performing the backward translation is different from the one that produced the forward translation.
- Distillation: The synthetic parallel data generated through this cross-model back-translation process is then used to train a separate, supervised MT model.
This process is designed to create more diverse and higher-quality synthetic data compared to standard iterative back-translation, where the same model is used for both forward and backward translation. CBD is presented as an add-on that can be applied to any existing UMT framework.
Datasets
The paper evaluates CBD on several standard UMT benchmarks:
- WMT'14 English-French (En-Fr)
- WMT'16 English-German (En-De)
- WMT'16 English-Romanian (En-Ro)
- IWSLT'13 English-French (En-Fr)
- IWSLT'14 English-German (En-De)
For training, they utilize large-scale monolingual news crawl datasets (e.g., 190M sentences for English in WMT). For some experiments, they also use smaller, default datasets (e.g., 5M sentences per language).
Key Insights and Ideas
- The Plateau of Diversification: The authors hypothesize that existing UMT diversification methods (like DAE and IBT) might be reaching their limits, leading to performance plateaus.
- Cross-Model is Key: The crucial insight is that using different UMT agents for the forward and backward translation steps in back-translation is essential for generating truly diverse data. This "cross-model" approach creates synthetic data that is more distinct from what a single agent would produce.
- Diversity Enhances Performance: CBD's success is attributed to its ability to embrace data diversity. Experiments show that variants that don't use this cross-model approach (e.g., BD(2,2)) do not yield the same improvements.
- Reconstruction BLEU as a Diversity Metric: Lower reconstruction BLEU scores (when translating and then translating back to the original language) indicate more diverse generated data, and CBD achieved significantly lower scores than its non-cross-model counterparts.
- Ensemble Distillation Fails in UMT: Ensemble knowledge distillation, which works well for supervised MT, performs poorly when applied to UMT agents. This is likely due to the inherent noise and potential for repetitions in the synthetic data generated by averaging UMT agent outputs.
Key Results
- State-of-the-Art Performance: CBD achieves state-of-the-art (SOTA) results on WMT'14 En-Fr (38.2 BLEU), WMT'16 En-De (30.1 BLEU), and WMT'16 En-Ro (36.3 BLEU) unsupervised translation tasks.
- Significant Improvements: CBD outperforms strong baselines like XLM by 1.5-3.3 BLEU on WMT and IWSLT tasks. For example, it improves XLM on WMT'14 En-Fr by 2.3 BLEU.
- Faster Convergence: Figure 2 shows that CBD converges much faster than the baseline XLM, requiring significantly fewer training updates to reach its peak performance.
- Robustness: CBD provides consistent gains even when using smaller datasets or when baselines are less powerful. It also shows improvements over PBSMT models, which are typically deterministic.
- Low Repetition: CBD generates synthetic data with very low tri-gram repetition rates (less than 0.1%), contrasting sharply with ensemble distillation methods that exhibit high repetition rates (up to 86%).
Limitations and Future Work
- Computational Cost: While CBD itself requires only one model for inference after training, the initial training of multiple UMT agents can be computationally intensive.
- Generalization of GCBD: The paper explores a generalized version (GCBD) with more than two agents, finding diminishing returns beyond a certain point, suggesting that simply increasing the number of agents isn't always optimal.
- Translationese Effect: The authors investigate the translationese effect and find that CBD performs well on natural source sentences, suggesting its practical usefulness. However, they note that the effect might be more pronounced in semi-supervised settings.
- Future Work: The paper mentions exploring more advanced techniques for pseudo-optimizing non-conditional language model losses and further investigating the translationese effect in unsupervised settings.
Summary updated