Modularized Multilingual NMT with Fine-grained Interlingua

Sungjun Lim, Yoonjung Choi, Sangha KimNAACL 2024 · DOI

TL;DR

This paper introduces a novel modularized multilingual Neural Machine Translation (NMT) architecture that enhances the generation of language-independent "interlingua" features. By employing a three-part encoder (source-specific, universal, and target-specific) and a new Denoising Bi-path Autoencoder (DBAE), the approach significantly improves zero-shot translation performance, especially for English-centric directions.

What Problem Does This Paper Address?

Multilingual Neural Machine Translation (MNMT) aims to train a single model for translating multiple language pairs, offering efficiency and scalability. A recent trend is "modularized MNMT," which uses language-specific encoders and decoders to manage model capacity. However, these modular approaches often struggle to produce truly language-independent features (interlingua), leading to degraded performance in zero-shot translation (translating between language pairs not explicitly seen during training). Previous attempts to address this by sharing upper encoder layers improved interlingua generation but didn't guarantee the propagation of language-specific features to the decoders. This paper tackles the challenge of creating robust interlingua features while ensuring effective language-specific information transfer for improved zero-shot translation.

Methods

The core innovation lies in a new modularized encoder architecture and a Denoising Bi-path Autoencoder (DBAE).

  • Modularized Encoder: The encoder is divided into three distinct modules:
  • Encs (Source Language-Specific): Handles input specific to the source language.
  • Encall (Universal): Shares parameters across all languages to learn language-independent features.
  • Enct (Target Language-Specific): Specifically designed to propagate target language features to the decoder. This structure aims to first generate interlingua features via Encall and then refine them with target language-specific information via Enct.

  • Denoising Bi-path Autoencoder (DBAE): To further improve zero-shot performance and address alignment issues, the authors propose DBAE. Unlike previous Denoising Autoencoders (DAE), DBAE uses a bi-path approach where the output of Encall is bifurcated and fed into both a target language-specific encoder (Enct') and a pivot language-specific encoder (Enct''). The outputs from these two paths are then weighted-summed before being passed to the decoder. This aims to align the encoder output with features preferred by non-pivot language decoders.

Datasets

The primary dataset used for training and evaluation is MultiUN. This dataset consists of parallel corpora for six languages (English, Arabic, Chinese, French, Russian, Spanish). The authors specifically use English-centric language pairs and create balanced (1:1:1) and unbalanced (1:2:4, 1:5:50) training settings to simulate different data distributions. For the DBAE task, monolingual corpora from the same dataset are utilized.

Key Insights and Ideas

  • The Importance of Enct: The introduction of a dedicated target language-specific encoder module (Enct) is crucial for refining interlingua features and improving translation performance, particularly in English-centric directions.
  • DBAE for Zero-Shot Alignment: The Denoising Bi-path Autoencoder (DBAE) significantly boosts zero-shot performance by effectively aligning encoder outputs with decoder expectations, especially for non-pivot languages. The bi-path mechanism, combining universal and pivot-specific features, proves more effective than a single-path DAE.
  • Balanced Feature Contribution: Ablation studies on DBAE show that equal weighting for the pivot and target language-specific paths (Enct' and Enct'') yields the best zero-shot performance, suggesting that both types of features are equally important for robust translation.
  • Explicit Language-Specific Features Matter: The results suggest that relying solely on universal features can lead to performance degradation, reinforcing the value of explicit language-specific feature propagation.

Key Results

  • The proposed modularized MNMT with Enct achieved an improved average BLEU score of +2.90 in En-to-Any directions and +3.06 in zero-shot settings compared to existing baselines.
  • The DBAE approach consistently outperformed other baselines in zero-shot scenarios across various data ratios.
  • The zero-shot hypothesis H₁ (single-path inference) performed better than H₂ (bi-path inference) in some evaluations, suggesting that simpler inference strategies can also be effective.
  • The model demonstrated significant improvements in En->Any directions but showed less pronounced gains in Any->En directions, indicating a potential area for future work.

Limitations and Future Work

  • Limited Language Scope: The experiments were conducted with only 5 languages, which the authors acknowledge is insufficient to definitively prove the optimal performance of their method for all language families.
  • Low-Resource Languages: The study did not account for low-resource languages or diverse domains, as it relied on simulated balanced/unbalanced settings and standard benchmarks.
  • Any->En Performance: While En->Any performance was strong, Any->En directions showed less improvement, suggesting a need for further decoder development or architectural enhancements.
  • Incremental Learning: The current model does not support incremental learning, which could be a valuable addition for handling new languages or data.
  • DBAE Expert Module: The DBAE's weighted summation uses fixed coefficients. Developing an "expert module" to learn these weights dynamically could further optimize performance.
  • Future Work: The authors plan to incorporate more languages through incremental learning, utilize more diverse evaluation benchmarks, and explore more complex bi-path structures (tri-path, quadri-path) for DBAE. They also aim to enhance Any->En performance.

Summary updated

BibTeX
@inproceedings{lim2024modularized,
  author = {Sungjun Lim and Yoonjung Choi and Sangha Kim},
  title = {{Modularized Multilingual NMT with Fine-grained Interlingua}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.328},
}
Figure