Abstract: Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization—a challenging setup that tests models on translation directions they have not been optimized for at training time. To solve the problem, we (i) reformulate multilingual translation as probabilistic inference, (ii) define the notion of zero-shot consistency and show why standard training often results in models unsuitable for zero-shot tasks, and (iii) introduce a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages. We test our multilingual NMT models on multiple public zero-shot translation benchmarks (IWSLT17, UN corpus, Europarl) and show that agreement-based learning often results in 2-3 BLEU zero-shot improvement over strong baselines without any loss in performance on supervised translation directions.
Introduction
Multilingual Neural Machine Translation (NMT) aims to build a single model capable of translating between multiple language pairs. A significant goal in this field is "zero-shot" translation: the ability to translate between a source and target language pair that was never explicitly seen together during training (e.g., French to German when only English-French and English-German data were provided). While shared-parameter models theoretically support this, practical zero-shot performance is often unstable and lags significantly behind supervised directions.
This discrepancy arises because standard training objectives, such as maximizing composite likelihood, do not explicitly coordinate how different languages map to a shared semantic space. Consequently, the model lacks a "consistency" guarantee; it may represent the same underlying meaning differently depending on the input language, leading to failures in zero-shot paths.
Consistency by Agreement
To address this, we propose a novel training objective: Consistency by Agreement. The core idea is to force the model to generate equivalent translations in an auxiliary language regardless of which language in a parallel pair is used as the source.
Agreement Objective
During training, if the model is given a parallel pair (e.g., an English sentence $x_{En}$ and its French translation $x_{Fr}$), we introduce a loss term that encourages the model to produce the same translation in a third "auxiliary" language (e.g., Spanish). This forces the encoders for different languages to align their internal representations into a shared "interlingual" space.
Technical Implementation
- Greedy Continuous Decoding: To maintain differentiability for backpropagation, we avoid discrete token selection. Instead, we employ a "soft" decoding process where output tokens are represented as weighted averages of vocabulary embeddings.
- Efficiency via Subsampling: To manage computational costs, we randomly sample one auxiliary language per training batch, keeping the overhead constant ($O(1)$) relative to the number of languages.
- Stop-Gradient Protection: We use a
stop_gradientoperator to ensure that the agreement loss primarily guides zero-shot directions without degrading the performance of supervised tasks.
Experimental Results
We evaluated the agreement-based method on three major benchmarks: the UN Corpus, Europarl v7, and IWSLT17. For IWSLT17, we introduced a "cleaned" version (IWSLT17*) to remove overlapping sentences that could artificially inflate zero-shot metrics.
As shown in the results, the agreement method consistently outperforms the "Basic" multilingual baseline by 2 to 3 BLEU points in zero-shot directions. Notably, our method achieves quality comparable to "pivoting" (Source $\rightarrow$ English $\rightarrow$ Target) but requires only a single direct translation pass, making it significantly more efficient at inference time. Furthermore, these gains in zero-shot performance do not compromise the accuracy of supervised translation directions.
Discussion and Conclusion
Our findings demonstrate that agreement-based learning provides a powerful regularizing signal, particularly when training data is scarce. By formalizing zero-shot consistency and enforcing internal alignment through auxiliary language agreement, we achieve more robust multilingual models.
While the method introduces some training overhead—roughly 4x on a single GPU—this is a one-time cost that can be mitigated through parallelization. Future work may explore applying this agreement framework to multi-modal tasks, ensuring semantic consistency across different data modalities like text and images.
Summary updated