Introduction
In the current landscape of Natural Language Processing, parameter-efficient fine-tuning (PEFT) techniques like prompt tuning have become the standard for adapting Large Language Models (LLMs) to specific tasks. By adding a small number of trainable vectors to the input, researchers can steer models like GPT-3 without updating billions of parameters. However, this success has not translated well to "small-sized" language models—those with fewer than 1 billion parameters, such as BERT or XLM-R.
For these smaller models, standard prompt tuning often yields poor performance, forcing a reliance on full fine-tuning. Unfortunately, full fine-tuning in low-resource or few-shot scenarios frequently leads to overfitting or "catastrophic forgetting," where the model loses its pre-trained multilingual capabilities while trying to learn a new task. This paper introduces Lottery Ticket Prompt-learning (LTP), a framework designed to bridge this gap and enable small models to excel in cross-lingual transfer.
The Challenge: Prompting at Scale
The primary difficulty in applying prompt tuning to small models lies in their limited capacity to absorb task-specific information through prompts alone. Unlike LLMs, which possess emergent properties that allow them to interpret soft prompts effectively, smaller models require more substantial architectural updates to adapt.
This challenge is amplified in cross-lingual settings, particularly for low-resource languages. When a model is fine-tuned on a tiny dataset for a language like Quechua or Guarani, it often memorizes the noise in the training data rather than learning the underlying linguistic structure. Existing solutions to this problem often involve complex "Deep Prompt Tuning" or require external linguistic resources like bilingual dictionaries, which are rarely available for indigenous or endangered languages.
The LTP Framework
The researchers propose a two-stage approach based on the Lottery Ticket Hypothesis, which suggests that dense neural networks contain sparse subnetworks ("winning tickets") that can reach the same level of performance as the original network when trained in isolation.
1. Identifying the Winning Ticket
The first stage involves finding the most "active" parameters within the model. The authors use a task-agnostic Masked Language Modeling (MLM) objective on a high-resource language (English Wikipedia). By measuring which parameters change the most during this process, they identify the subnetwork most vital for learning. These parameters constitute the "winning ticket."
2. Sparse Prompt Fine-tuning
Once the winning ticket is identified, the model undergoes task-specific training. In this stage, the framework prepends "soft prompts" (trainable continuous vectors) to the input. Crucially, the training process only updates the soft prompt parameters and the identified winning ticket subset (typically 20% to 75% of the model). The remaining majority of the model's parameters are frozen. This sparse update mechanism allows the model to adapt to new tasks and languages without destroying the foundational multilingual structures established during pre-training.
Experimental Results and Insights
The LTP framework was evaluated using the XNLI benchmark (15 languages) and the AmericasNLI benchmark, which focuses on 10 low-resource indigenous languages of the Americas. The experiments focused on few-shot scenarios, ranging from 1 to 32 examples per class.
Key Findings
- The Middle Layer Advantage: The researchers discovered that the "active" parameters in the middle layers (specifically layers 5-8 of a 12-layer model) are the most expressive for cross-lingual transfer. Tuning these layers provides a better balance between task adaptation and knowledge preservation than tuning the early embedding layers or the final output layers.
- Sparsity as a Regularizer: Updating only 20% of the model’s parameters frequently outperformed full 100% fine-tuning. This suggests that for small models, the "noise" introduced by full fine-tuning on small datasets is more detrimental than the constraints of sparse tuning. LTP effectively acts as a regularizer, preventing the model from overfitting to the few-shot examples.
- Cross-lingual Robustness: Despite identifying the winning tickets using English data, the tickets proved highly effective for transferring knowledge to unrelated, low-resource languages. On the AmericasNLI dataset, LTP using only 64 samples per class outperformed models that had been fine-tuned on massive datasets of 390,000 samples.
Conclusion
Lottery Ticket Prompt-learning (LTP) provides a robust path forward for utilizing small-sized language models in complex cross-lingual tasks. By combining the parameter efficiency of prompt tuning with the targeted updates of the Lottery Ticket Hypothesis, LTP prevents the catastrophic forgetting typically associated with fine-tuning small models. This framework is particularly valuable for low-resource language applications, where data scarcity makes traditional training methods unfeasible. Future work may explore more sophisticated parameter selection methods and the applicability of specific winning tickets to different task types, such as generative versus discriminative objectives.
Summary updated