Abstract
Yining Wang, Jiajun Zhang, Long Zhou, Yuchen Liu, Chengqing Zong. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
In traditional Multilingual Neural Machine Translation (MNMT), models are typically trained to translate from a source language to a single target language at a time. Even when these models are capable of handling multiple target languages, they generally generate them independently during inference. This approach fails to leverage the potential synergy between different target languages, which often contain complementary structural and lexical information.
Methods: SyncTrans and Interactive Decoding
The authors introduce SyncTrans, a framework that enables the synchronous generation of two target languages. This is achieved through two key components:
- Interactive Decoding Algorithm: Tokens for both target languages are predicted in parallel at each time step, allowing the decoders to "talk" to each other during generation.
- Synchronous Attention Model (SyncAtt): This architecture modifies the standard Transformer decoder. Instead of only attending to its own previously generated tokens, each decoder also attends to the history of the "partner" language. These two attention vectors are combined using a gated function to create a unified hidden state.
To address the lack of trilingual training data, the researchers utilized data augmentation to create "pseudo-trilingual" datasets from standard bilingual corpora by using pre-trained models to fill in missing translations.
Cross-Lingual Synergy
A major insight of this work is that languages with divergent word orders, such as Chinese (SVO) and Japanese (SOV), benefit significantly from synchronous decoding. For example, the Chinese decoder may predict a verb earlier in the sequence than the Japanese decoder. By sharing this information, the Japanese decoder can use the already-predicted Chinese verb as a "look-ahead" context to resolve lexical ambiguities and structural hurdles.
Key Results
The SyncTrans model was evaluated on the IWSLT (TED Talks) and WMT14 benchmarks using BLEU scores. The results demonstrate that the interactive decoding approach consistently outperforms both individual bilingual models and standard multilingual models that lack interaction.
On the English-to-Japanese task, SyncTrans improved the BLEU score by 2.75 points over the individual baseline and approximately 1.0 point over a standard multilingual model. These improvements were also observed on the larger WMT14 dataset, where SyncTrans outperformed the baseline by 0.5 to 0.7 BLEU points, confirming the scalability of the approach.
Limitations and Future Work
While SyncTrans shows clear benefits for bilingual generation, scaling the architecture to $N$ languages presents challenges in computational complexity, as the number of attention paths grows quadratically. Additionally, the model's performance is sensitive to the quality of the augmented trilingual data. Future work will focus on more efficient interaction mechanisms and exploring the framework's applicability to a broader range of language families.
Summary updated