Abstract
Adapting multilingual pre-trained language models (mPLMs) to languages absent from their initial training data typically requires resource-intensive full-model fine-tuning. This is often impractical for low-resource languages due to data scarcity and computational costs. This paper introduces an efficient two-stage soft-prompt tuning framework that adapts models using only 0.28% of the original parameters. Our approach outperforms traditional fine-tuning and matches state-of-the-art adapter methods while being significantly more storage-efficient, making it ideal for cross-lingual transfer to unseen languages.
Introduction
Multilingual Pre-trained Language Models (mPLMs) like XLM-R are the foundation of modern cross-lingual NLP, excelling at zero-shot transfer between languages included in their pre-training. However, they struggle with "unseen languages." The standard solution—continued pre-training on the whole model—is problematic because tuning billions of parameters is computationally expensive and can lead to "catastrophic forgetting" of general knowledge when data is scarce.
Methodology
We propose a two-stage adaptation strategy using soft prompts—trainable vectors prepended to the input that guide the frozen model's state without modifying its core weights.
- Stage 1: Language Adaptation: The model is trained on unlabeled text from both a source language (e.g., English) and the unseen target language using a Masked Language Modeling (MLM) objective. Soft prompts are tuned across all layers to learn the target language's linguistic characteristics.
- Stage 2: Task Adaptation: The model is trained on a specific downstream task using labeled source-language data.
The Top-K Strategy
A critical component of our method is the "Top-K" strategy. During task adaptation, we only tune soft prompts in the upper layers. This design is based on the observation that lower layers in mPLMs typically handle language-specific features, while upper layers are more task-focused and language-independent. Freezing the lower-layer prompts preserves the language knowledge acquired in Stage 1.
Experimental Setup
We evaluated our method on two challenging benchmarks for low-resource, unseen languages:
- MasakhaNEWS: News topic classification for 8 African languages (e.g., Igbo, Lingala, Luganda).
- AmericasNLI: Natural Language Inference for 10 Indigenous languages of the Americas (e.g., Quechua, Guarani). Unlabeled data for these languages was sourced from repositories like AmericaNLP, with sizes ranging from 0.6K to 125K sentences.
Results and Discussion
Our soft-prompt tuning method demonstrated superior performance and efficiency across multiple dimensions:
- Performance: On MasakhaNEWS, our method achieved an average accuracy of 82.10%, surpassing full fine-tuning (80.78%) despite using a fraction of the parameters.
- Parameter Efficiency: By tuning only 1.57M parameters (0.28% of XLM-R Large), the model avoids the instability often seen in full-parameter tuning on small datasets.
- Layer Specialization: Empirical results support the Top-K strategy. As shown in Figure 6, tuning the bottom layers during task adaptation significantly degrades performance compared to tuning the top layers, confirming that language-specific information is best preserved in the lower layers.
- Storage Efficiency: A language-specific checkpoint for our method is only 6.2MB, compared to 2.24GB for a full fine-tuned model. This allows for the deployment of a single base model that can serve dozens of languages simultaneously.
Conclusion
Soft-prompt tuning provides a highly efficient and robust pathway for adapting mPLMs to unseen, low-resource languages. While the method effectively addresses parameter and data efficiency, it remains constrained by the fixed vocabulary of the base mPLM. Future research into byte-level models may offer a solution to this tokenizer bottleneck, further extending the reach of multilingual NLP to all the world's languages.
Summary updated