Abstract
Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to 0.4% for high and medium-resource languages and 1.6% for low-resource ones.
Introduction
Multilingual Neural Machine Translation (MNMT) aims to develop unified models capable of translating across hundreds of language pairs. While pre-trained models like M2M-100 provide a strong foundation, fine-tuning them for specific language sets presents significant challenges. Standard full-parameter fine-tuning is computationally expensive and often suffers from "negative interference," where the shared parameter space leads to performance degradation in high-resource languages as the model attempts to accommodate low-resource ones.
This paper introduces Language-Specific LoRA (LSLo), a framework designed to isolate and optimize the "intrinsic subspaces" required for each language. By updating only a tiny fraction of the model's parameters, LSLo improves translation quality while significantly reducing the computational footprint.
Proposed Method: LSLo and GPS
The proposed approach consists of three main components:
- Language-Specific LoRA (LSLo) Modules: Instead of a single shared adapter, the model employs multiple sparsely activated LoRA modules. During a forward pass, only the module corresponding to the specific source or target language is activated, effectively isolating language-specific knowledge.
- Weight Learning Gate: To determine whether a layer should focus on the source or target language, the authors use a differentiable weight learning gate. This removes the need for manual architectural tuning and allows the model to automatically discover the optimal indexing strategy.
- Gradual Pruning Schedule (GPS): To identify the minimal necessary parameters (the intrinsic subspace), the authors implement a pruning schedule. Training begins with a larger parameter set which is gradually reduced to a target sparsity, ensuring stability and performance.
Analysis of Language Subspaces
The researchers' analysis yielded several key insights into the internal mechanics of MNMT models:
The Encoder-Decoder Shift
The weight learning gate revealed a distinct architectural trend. As shown in the figure below, the model primarily focuses on the source language in the initial layers of the encoder. As processing moves through the top encoder layers and into the decoder, the focus shifts decisively toward the target language.
FFN vs. Attention
Contrary to many PEFT methods that prioritize attention layers, this study found that applying LSLo to the Feed-Forward Network (FFN) layers (specifically fc1 and fc2) is more effective for capturing language-specific nuances. This suggests that the FFN components act as the primary repository for language-specific knowledge in these models.
Experimental Results
The method was evaluated on the FLORES-101 benchmark using 12-language (lang12) and 30-language (lang30) subsets, with M2M-124 (615M) as the base model.
Performance and Efficiency
LSLo demonstrated superior performance across various resource tiers (High, Medium, Low). Notably, it mitigated the performance drop typically seen in high-resource languages during multilingual fine-tuning. While standard fine-tuning often leads to a decline (e.g., -2.47 spBLEU), LSLo achieved an increase of +1.37 spBLEU.
As detailed in the table above, the LSLo + GPS approach achieved the best average performance (17.70 spBLEU on the 12-language subset), outperforming the full-parameter fine-tuning baseline (15.43 spBLEU) while using significantly fewer trainable parameters—as low as 0.4% for high-resource languages.
Conclusion
This work demonstrates that multilingual fine-tuning can be made both more effective and more efficient by targeting intrinsic language-specific subspaces. By isolating these subspaces through LSLo and optimizing them via architecture learning and gradual pruning, researchers can achieve state-of-the-art translation performance with minimal parameter updates. Future work will explore scaling these methods to even larger models and a broader array of languages.
Summary updated