Abstract
The many-to-many multilingual neural machine translation can be regarded as the process of integrating semantic features from the source sentences and linguistic features from the target sentences. To enhance zero-shot translation, models need to share knowledge across languages, which can be achieved through auxiliary tasks for learning a universal representation or cross-lingual mapping. To this end, we propose to exploit both semantic and linguistic features between multiple languages to enhance multilingual translation. On the encoder side, we introduce a disentangling learning task that aligns encoder representations by disentangling semantic and linguistic features, thus facilitating knowledge transfer while preserving complete information. On the decoder side, we leverage a linguistic encoder to integrate low-level linguistic features to assist in the target language generation. Experimental results on multilingual datasets demonstrate significant improvement in zero-shot translation compared to the baseline system, while maintaining performance in supervised translation. Further analysis validates the effectiveness of our method in leveraging both semantic and linguistic features.
1. Introduction
Multilingual Neural Machine Translation (MNMT) aims to facilitate translation across multiple language pairs within a single unified model. While MNMT models perform well on language pairs seen during training, they often struggle with "zero-shot" translation—pairs for which no parallel data was provided. This challenge stems from the entanglement of language-agnostic semantic meaning with language-specific linguistic structures in the encoder. This entanglement leads to a representation gap between languages and an "off-target" generation problem, where the decoder produces text in an incorrect language.
2. Methodology
We propose a novel architecture designed to explicitly disentangle "what is being said" (semantics) from "how it is structured" (linguistics).
2.1 Encoder-Side Disentanglement
The model introduces a Disentangler module following the standard encoder. It utilizes two parallel Feed-Forward Networks (FFNs):
- Semantic FFN: Extracts language-agnostic features using contrastive learning to align parallel sentence pairs in a shared space.
- Language FFN: Captures language-specific features by grouping sentences of the same language together. A reconstruction loss is employed to ensure that the combination of these features can perfectly reconstruct the original encoder output, making the disentanglement process "lossless."
2.2 Decoder-Side Linguistic Guidance
To mitigate off-target issues, we introduce a shallow Linguistic Encoder (2 layers) on the decoder side. This module extracts low-level linguistic features from the target language, which are then fused with high-level semantic features from the encoder. This provides the decoder with explicit linguistic "guardrails" to guide the generation process toward the correct target language.
3. Experiments and Results
The proposed method was evaluated on three multilingual benchmarks: IWSLT2017, OPUS-7, and PC-6.
3.1 Zero-Shot Translation Performance
Our approach achieved a substantial average improvement of +3.74 BLEU points across all zero-shot directions compared to the baseline m-Transformer. Importantly, this was achieved without degrading performance in supervised directions, maintaining the stability of the translation system.
3.2 Mitigation of Off-Target Issues
The inclusion of explicit linguistic features significantly reduced the off-target translation rate. On the IWSLT2017 dataset, the rate dropped from 6.76% to 2.46%, while on the more challenging PC-6 dataset, it plummeted from 21.02% to 7.25%.
3.3 Efficiency and Scale
Analysis shows that our task-specific architecture (approx. 100M parameters) can outperform much larger models like Alpaca-7B and remain competitive with ChatGPT in zero-shot tasks, utilizing only a fraction (0.1%) of the parameters.
4. Conclusion
By explicitly disentangling semantic and linguistic features, we significantly enhance the zero-shot capabilities of MNMT models. The proposed architecture effectively bridges the representation gap and provides the decoder with the necessary linguistic guidance to avoid off-target generation. Future work will investigate the scalability of this approach to massively multilingual environments and the incorporation of more explicit linguistic priors.
Summary updated