Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation

Zhe Cao, Zhi Qu, Hidetaka Kamigaito, Taro Watanabe

Abstract

Multilingual neural machine translation (MNMT) models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters is often inefficient and can lead to negative interactions among languages. In this work, we demonstrate that fine-tuning for a specific language occurs in its intrinsic language-specific subspace, requiring only a tiny fraction of the total parameters. We propose Language-Specific LoRA (LSLo) to isolate these subspaces, alongside architecture learning techniques and a Gradual Pruning Schedule (GPS) to identify the optimal minimal subspaces for each language. Experimental results on FLORES-101 subsets show that our methods outperform full-parameter fine-tuning by up to 2.25 spBLEU while reducing trainable parameters to as little as 0.4%.

Introduction

Multilingual Neural Machine Translation (MNMT) models are powerful because they consolidate the translation capabilities of hundreds of languages into a single architecture. However, adapting these massive models to specific datasets or language pairs presents two major hurdles. First, inefficiency: updating all parameters (full-parameter fine-tuning) is computationally expensive as models scale into the billions of parameters. Second, negative interference: languages often "compete" for the same parameter space. Ironically, standard fine-tuning often decreases performance for high-resource languages because the model over-fits or suffers from updates meant for other languages.

This research addresses these challenges by hypothesizing that each language only needs a small, specific subset of the model's capacity—an intrinsic subspace—to be fine-tuned effectively.

Methodology

The authors introduce several optimization techniques to isolate and exploit these subspaces:

Language-Specific LoRA (LSLo)

Instead of sharing a single Low-Rank Adaptation (LoRA) module across all languages, LSLo utilizes multiple modules. During a forward pass, only the module corresponding to the specific source or target language is activated. This ensures that language-specific knowledge is stored in dedicated parameters, reducing cross-language interference.

Weight Learning and Architecture Optimization

To avoid manually determining whether a specific layer should focus on the source or target language, the researchers employed a "Weight Learning" technique. This automatically learns the optimal indexing strategy (source-indexed vs. target-indexed) for each layer in the Transformer architecture.

Gradual Pruning Schedule (GPS)

To find the absolute minimum number of parameters required for each language, the authors applied unstructured pruning during the fine-tuning process. The GPS starts at zero pruning and gradually increases to a target ratio (up to 90%). This allows the model to stabilize its performance while shedding unnecessary weights, effectively distilling the fine-tuning process into the most critical parameters.

Experimental Setup

The study utilized the FLORES-101 benchmark, a high-quality parallel dataset. The researchers tested two subsets:

  • lang12: 12 languages representing High (H), Medium (M), and Very-Low (V) resource levels.
  • lang30: A larger subset of 30 languages to test scalability.
  • Base Model: The experiments were conducted using the M2M-124 (615M parameters) model.

Key Insights

Resource Level vs. Subspace Size

There is a strong correlation between a language's data availability and its required subspace size. High-resource languages can be fine-tuned effectively in tiny subspaces (e.g., rank 2), whereas low-resource languages benefit from larger subspaces (rank 8 or higher) to capture necessary nuances.

Mitigating the Over-fitting Trap

The decline in high-resource language performance during standard fine-tuning is largely due to over-fitting. By restricting these languages to a very small subspace and applying aggressive pruning, the researchers transformed a 2.47 spBLEU loss into a 1.37 spBLEU gain.

Structural Importance: FFN vs. Attention

The research found that Feed-Forward Network (FFN) layers (specifically fc1 and fc2) are far more critical for storing language-specific knowledge than attention mechanisms. Concentrating parameters in the FFN layers yielded better results than spreading them across the entire Transformer block.

Encoder-Decoder Transition

Weight learning revealed a distinct structural pattern: the bottom layers of the encoder focus primarily on the source language, while the top layers of the encoder and the entire decoder focus almost exclusively on the target language.

Results and Conclusion

The LSLo + GPS approach demonstrated significant improvements in both efficiency and translation quality:

  • Performance: On the lang30 subset, the method outperformed full-parameter fine-tuning by 2.25 spBLEU points.
  • Efficiency: Trainable parameters were reduced to 0.4% for high and medium-resource languages and 1.6% for low-resource ones.

While the current work groups languages into broad resource categories, future research may involve setting individual ranks and pruning ratios for every language. Additionally, testing these intrinsic subspaces on models exceeding 10 billion parameters remains a promising avenue for scaling lightweight fine-tuning.

Summary updated

Abstract

Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to $0.4\%$ for high and medium-resource languages and $1.6\%$ for low-resource ones.
BibTeX
@inproceedings{cao2024exploring,
  author = {Zhe Cao and Zhi Qu and Hidetaka Kamigaito and Taro Watanabe},
  title = {{Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=hZDSxPChHa},
}
Figure