Abstract
Multilingual neural machine translation (MNMT) models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters is often inefficient and can lead to negative interactions among languages. In this work, we demonstrate that fine-tuning for a specific language occurs in its intrinsic language-specific subspace, requiring only a tiny fraction of the total parameters. We propose Language-Specific LoRA (LSLo) to isolate these subspaces, alongside architecture learning techniques and a Gradual Pruning Schedule (GPS) to identify the optimal minimal subspaces for each language. Experimental results on FLORES-101 subsets show that our methods outperform full-parameter fine-tuning by up to 2.25 spBLEU while reducing trainable parameters to as little as 0.4%.
Introduction
Multilingual Neural Machine Translation (MNMT) models are powerful because they consolidate the translation capabilities of hundreds of languages into a single architecture. However, adapting these massive models to specific datasets or language pairs presents two major hurdles. First, inefficiency: updating all parameters (full-parameter fine-tuning) is computationally expensive as models scale into the billions of parameters. Second, negative interference: languages often "compete" for the same parameter space. Ironically, standard fine-tuning often decreases performance for high-resource languages because the model over-fits or suffers from updates meant for other languages.
This research addresses these challenges by hypothesizing that each language only needs a small, specific subset of the model's capacity—an intrinsic subspace—to be fine-tuned effectively.
Methodology
The authors introduce several optimization techniques to isolate and exploit these subspaces:
Language-Specific LoRA (LSLo)
Instead of sharing a single Low-Rank Adaptation (LoRA) module across all languages, LSLo utilizes multiple modules. During a forward pass, only the module corresponding to the specific source or target language is activated. This ensures that language-specific knowledge is stored in dedicated parameters, reducing cross-language interference.
Weight Learning and Architecture Optimization
To avoid manually determining whether a specific layer should focus on the source or target language, the researchers employed a "Weight Learning" technique. This automatically learns the optimal indexing strategy (source-indexed vs. target-indexed) for each layer in the Transformer architecture.
Gradual Pruning Schedule (GPS)
To find the absolute minimum number of parameters required for each language, the authors applied unstructured pruning during the fine-tuning process. The GPS starts at zero pruning and gradually increases to a target ratio (up to 90%). This allows the model to stabilize its performance while shedding unnecessary weights, effectively distilling the fine-tuning process into the most critical parameters.
Experimental Setup
The study utilized the FLORES-101 benchmark, a high-quality parallel dataset. The researchers tested two subsets:
- lang12: 12 languages representing High (H), Medium (M), and Very-Low (V) resource levels.
- lang30: A larger subset of 30 languages to test scalability.
- Base Model: The experiments were conducted using the M2M-124 (615M parameters) model.
Key Insights
Resource Level vs. Subspace Size
There is a strong correlation between a language's data availability and its required subspace size. High-resource languages can be fine-tuned effectively in tiny subspaces (e.g., rank 2), whereas low-resource languages benefit from larger subspaces (rank 8 or higher) to capture necessary nuances.
Mitigating the Over-fitting Trap
The decline in high-resource language performance during standard fine-tuning is largely due to over-fitting. By restricting these languages to a very small subspace and applying aggressive pruning, the researchers transformed a 2.47 spBLEU loss into a 1.37 spBLEU gain.
Structural Importance: FFN vs. Attention
The research found that Feed-Forward Network (FFN) layers (specifically fc1 and fc2) are far more critical for storing language-specific knowledge than attention mechanisms. Concentrating parameters in the FFN layers yielded better results than spreading them across the entire Transformer block.
Encoder-Decoder Transition
Weight learning revealed a distinct structural pattern: the bottom layers of the encoder focus primarily on the source language, while the top layers of the encoder and the entire decoder focus almost exclusively on the target language.
Results and Conclusion
The LSLo + GPS approach demonstrated significant improvements in both efficiency and translation quality:
- Performance: On the
lang30subset, the method outperformed full-parameter fine-tuning by 2.25 spBLEU points. - Efficiency: Trainable parameters were reduced to 0.4% for high and medium-resource languages and 1.6% for low-resource ones.
While the current work groups languages into broad resource categories, future research may involve setting individual ranks and pruning ratios for every language. Additionally, testing these intrinsic subspaces on models exceeding 10 billion parameters remains a promising avenue for scaling lightweight fine-tuning.
Summary updated