Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features

Mengyu Bu, Shuhao Gu, Yang Feng

Abstract

The many-to-many multilingual neural machine translation (MNMT) can be regarded as the process of integrating semantic features from source sentences and linguistic features from target sentences. To enhance zero-shot translation, models need to share knowledge across languages, which can be achieved through auxiliary tasks for learning a universal representation or cross-lingual mapping. To this end, we propose to exploit both semantic and linguistic features between multiple languages to enhance multilingual translation. On the encoder side, we introduce a disentangling learning task that aligns encoder representations by disentangling semantic and linguistic features, thus facilitating knowledge transfer while preserving complete information. On the decoder side, we leverage a linguistic encoder to integrate low-level linguistic features to assist in the target language generation. Experimental results on multilingual datasets demonstrate significant improvement in zero-shot translation compared to the baseline system, while maintaining performance in supervised translation. Further analysis validates the effectiveness of our method in leveraging both semantic and linguistic features. The code is available at https://github.com/ictnlp/SemLing-MNMT.

1. Introduction

Multilingual Neural Machine Translation (MNMT) aims to handle many languages within a single model, offering significant efficiency gains. However, a major hurdle remains: zero-shot translation. This occurs when a model is tasked with translating between a language pair it never encountered during training. The primary cause of poor zero-shot performance is "feature entanglement," where standard models mix universal semantic meaning with language-specific structural traits. This leads to encoder confusion and "off-target" issues, where the decoder outputs text in the wrong language due to a lack of clear linguistic guidance.

2. Proposed Method: SemLing-MNMT

We propose a two-pronged architecture designed to "untangle" the representation space by explicitly separating semantics from linguistics.

2.1 The Disentangler (Encoder Side)

Instead of a standard feed-forward network, we introduce a Disentangler module. It consists of two parallel paths:

  • Semantic FFN: Employs contrastive learning to pull parallel sentences (e.g., "Hello" and "Bonjour") closer together in the vector space, regardless of language.
  • Language FFN: Uses contrastive learning to push different languages apart, focusing on language-specific patterns.
  • Reconstruction Loss: To ensure no information is lost, a reconstruction loss ensures that the sum of semantic and linguistic vectors recreates the original encoder output.

2.2 Linguistic Guidance (Decoder Side)

To prevent the model from drifting into the wrong language, we introduce a Linguistic Encoder. This module extracts low-level features—which naturally contain more structural information—and feeds them into a Fusion Layer alongside high-level semantic features. This provides the decoder with a clear "linguistic roadmap" for the target language generation.

The SemLing-MNMT architecture featuring the Disentangler and Linguistic Encoder.

3. Experimental Evaluation

3.1 Datasets

The model was evaluated on three major multilingual benchmarks: IWSLT2017 (5 languages, 12 zero-shot directions), OPUS-7 (7 languages, 30 zero-shot directions), and PC-6 (6 languages including low-resource ones like Kazakh, 20 zero-shot directions).

3.2 Key Results

The proposed SemLing-MNMT showed impressive gains over standard Transformer baselines and state-of-the-art methods like mRASP2:

  • Zero-Shot Boost: Achieved an average improvement of +3.74 BLEU points in zero-shot directions across all datasets.
  • Reduced Off-Target Rate: On the PC-6 dataset, the off-target rate dropped from 21.02% to just 7.25%.
  • Efficiency vs. Scale: Despite having only ~100M parameters, the model outperformed Alpaca-7B and achieved over 90% of the performance of ChatGPT (GPT-3.5) in zero-shot tasks.

4. Analysis and Insights

4.1 Preservation over Elimination

Previous research often attempted to make encoders "language-agnostic" by stripping away language information. We argue that language-specific info is useful; it simply needs to be isolated so it does not clutter semantic meaning.

4.2 Visualizing the Split

t-SNE visualizations confirm the effectiveness of the Disentangler. The Semantic FFN successfully clusters sentences by meaning (mixing languages), while the Language FFN clusters them strictly by language, as shown in the distinct color blocks below.

t-SNE visualization of the Language FFN output, showing distinct clustering by language.

5. Conclusion

SemLing-MNMT significantly improves zero-shot translation by disentangling semantic and linguistic features. While supervised gains are modest (+0.18 BLEU), the reduction in off-target errors and the boost in zero-shot performance demonstrate the value of explicit feature modeling. Future work will explore scaling this approach to massively multilingual datasets and explicitly modeling syntax and morphology.

Summary updated

Abstract

The many-to-many multilingual neural machine translation can be regarded as the process of integrating semantic features from the source sentences and linguistic features from the target sentences. To enhance zero-shot translation, models need to share knowledge across languages, which can be achieved through auxiliary tasks for learning a universal representation or cross-lingual mapping. To this end, we propose to exploit both semantic and linguistic features between multiple languages to enhance multilingual translation. On the encoder side, we introduce a disentangling learning task that aligns encoder representations by disentangling semantic and linguistic features, thus facilitating knowledge transfer while preserving complete information. On the decoder side, we leverage a linguistic encoder to integrate low-level linguistic features to assist in the target language generation. Experimental results on multilingual datasets demonstrate significant improvement in zero-shot translation compared to the baseline system, while maintaining performance in supervised translation. Further analysis validates the effectiveness of our method in leveraging both semantic and linguistic features. The code is available at https://github.com/ictnlp/SemLing-MNMT.
BibTeX
@inproceedings{bu2024improving,
  author = {Mengyu Bu and Shuhao Gu and Yang Feng},
  title = {{Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=fEinc6XAn8},
}
Figure