Abstract
Maruan Al-Shedivat, Ankur Parikh. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
2019.
Introduction
The "holy grail" of multilingual Neural Machine Translation (NMT) is zero-shot translation—the ability to translate between language pairs that were never seen together during training. While theoretically possible if a model learns a language-agnostic "interlingua," standard architectures often fail to generalize. This failure stems from a lack of alignment in the latent space; representations of the same meaning in different languages do not naturally converge without explicit grounding. Consequently, zero-shot paths remain unoptimized "dark alleys," leading to poor performance and off-target translations.
Methods: Agreement-based Learning
To address this, Al-Shedivat and Parikh propose a training method centered on a consistency constraint. The core idea is to force the model to "agree" with itself: it must produce the same translation in an auxiliary language regardless of the source language used.
- The Agreement Objective: For a parallel sentence pair in English ($En$) and French ($Fr$), the model is tasked with translating both into a third language, such as German ($De$). An "agreement loss" is applied to penalize discrepancies between these two generated translations.
- Probabilistic Reformulation: The authors frame this consistency as a probabilistic inference problem. By maximizing the likelihood of agreement on translations into auxiliary languages, they provide a mathematical foundation for zero-shot consistency. They argue that if a model is consistent—producing the same output for the same underlying meaning regardless of input language—it is inherently more likely to generalize to unseen language pairs.
- Differentiable Decoding: A significant challenge in training this objective is the discrete nature of text generation, which typically prevents backpropagation. The authors overcome this by employing greedy continuous decoding. Instead of hard word selection, they use soft representations—weighted averages of word embeddings based on softmax outputs—to maintain differentiability throughout the agreement loop.
- Efficiency and Stability: To manage computational complexity, the model randomly subsamples one auxiliary language per training step. Furthermore, a
stop_gradientoperator is utilized to ensure that the agreement signal optimizes zero-shot paths without interfering with the established quality of supervised translation directions.
Evaluation and Results
The approach was evaluated on several major benchmarks, including the UN Corpus, Europarl v7, and a "cleaned" version of IWSLT17. The latter was specifically designed to test true zero-shot generalization by strictly removing overlapping sentences that might inflate scores in standard multilingual setups.
The results demonstrate that agreement-based learning consistently outperforms standard multilingual baselines by 2-3 BLEU points across nearly all zero-shot directions. Notably, these gains do not come at the expense of supervised task quality; performance on trained pairs like $En \to Fr$ remained stable. The model achieves performance comparable to "pivoting" (a two-step translation process) while maintaining the speed and simplicity of a single end-to-end model.
The method also proves particularly robust in low-data regimes. While standard models often degrade significantly with fewer than 100,000 parallel sentences, the agreement-based approach maintains high translation quality, suggesting that the consistency constraint acts as a powerful regularizer that guides the model toward more universal representations.
Conclusion
By incorporating a simple yet mathematically grounded consistency constraint, this work significantly improves the stability and performance of zero-shot NMT. While the method introduces some computational overhead during training—roughly 4x slower due to the extra encoding/decoding steps—the resulting models are more robust and better aligned. Future work may extend this agreement framework beyond translation to multimodal tasks, ensuring consistency between representations of different data types, such as images and text descriptions.
Summary updated