Abstract
While prompt-based finetuning has emerged as a superior paradigm for few-shot learning, its efficacy in full-data cross-lingual scenarios remains less understood. This paper investigates the performance of the PROFIT (PROmpt-based FInuetuning for cross-lingual Transfer) pipeline across various Multilingual Pretrained Language Models (MPLMs) and tasks. Our findings reveal that while prompting generally outperforms vanilla finetuning, its advantages are modulated by task complexity, model scale, and the linguistic proximity of target languages to the source.
Introduction
In the evolution of Multilingual Pretrained Language Models (MPLMs) such as mBERT and XLM-R, the standard approach for downstream task adaptation has been Vanilla Finetuning. This method typically involves appending a task-specific classifier head to the [CLS] token and training the model on labeled data. Recently, Prompt-Based Finetuning has challenged this convention. By reformulating tasks as "cloze-style" queries (e.g., converting a sentiment task into "This movie was [MASK]"), prompting allows models to leverage their original pretraining objectives more effectively.
While the benefits of prompting in data-scarce (few-shot) environments are well-documented, its performance in full-data regimes and its robustness during cross-lingual transfer—where a model is trained in one language and tested in others—have remained under-explored. This study evaluates whether prompting is a universal improvement or if it faces diminishing returns in large-scale multilingual contexts.
Methodology: The PROFIT Pipeline
The researchers introduce the PROFIT pipeline, which centers on the use of Pattern-Verbalizer Pairs (PVPs):
- Pattern: A template that transforms standard input into a prompt (e.g., "Input: [Sentence] Question: Is this a paraphrase? Answer: [MASK]").
- Verbalizer: A semantic mapping that translates discrete class labels into concrete vocabulary tokens (e.g., mapping Label 1 to "Yes" and Label 0 to "No").
By using actual words instead of arbitrary numerical labels, the model can tap into the semantic knowledge acquired during pretraining. The study compares PROFIT against Vanilla Finetuning and a zero-shot "Direct" baseline using mBERT and XLM-R. Experiments were conducted across a data spectrum ranging from $K=1$ (one example per class) to the full available dataset.
Experimental Design and Datasets
The evaluation spans three core Cross-Lingual Language Understanding (XLU) tasks:
- Amazon Reviews: Multiclass sentiment analysis (1-5 stars) across 6 languages.
- PAWS-X: Binary paraphrase identification across 7 languages.
- XNLI: Natural Language Inference (entailment, neutral, contradiction) across 15 languages.
Key Insights
1. The Complexity Delay
The advantage of prompting is not uniform across task types. For simple tasks like sentiment analysis, the most significant gains occur at the $K=1$ level. However, for complex sentence-pair tasks like NLI or paraphrase detection, the performance gap between prompting and vanilla finetuning actually widens as more data is introduced, before eventually stabilizing. This suggests that for complex logical relationships, the model requires a baseline of examples to effectively "anchor" the logic of the prompt.
2. Model Saturation and Scale
Model architecture significantly influences the utility of prompting. The performance delta is notably larger for mBERT than for XLM-R. Because XLM-R is a larger model trained on more extensive data, its vanilla finetuning performance is already high, leaving less "headroom" for prompting to provide substantial improvements in full-data scenarios.
3. Linguistic Gravity and Pretraining
Prompting does not equalize performance across all languages. Cross-lingual transfer success remains heavily dependent on language similarity (proximity to English) and the pretraining corpus size for the target language. If a language was underrepresented during the model's initial pretraining phase, the semantic cues provided by a prompt are less effective at bridging the transfer gap.
Results and Performance
- Full-Data Performance: PROFIT consistently outperformed Vanilla Finetuning. On average, PROFIT-mBERT yielded a 1.23% improvement, while PROFIT-XLM-R showed a more modest gain of 0.18%.
- Few-Shot Dominance: The most dramatic results were observed in the $K=1$ sentiment analysis setting, where PROFIT-XLM-R achieved 33.63% accuracy, significantly outperforming the 21.71% achieved by Vanilla Finetuning.
- Statistical Significance: T-tests confirmed that for mBERT, the improvements from prompting are statistically significant across all tasks. For XLM-R, the gains are highly significant in few-shot settings but become marginal as the dataset size increases.
Conclusion and Future Directions
The PROFIT pipeline demonstrates that prompt-based finetuning is a robust alternative to vanilla methods, particularly when leveraging the semantic meaning of labels. However, its "superpowers" are most evident in smaller models and data-constrained environments.
Future research should expand these insights into sequence labeling tasks (such as NER) and explore automated or "soft" prompt generation to further optimize cross-lingual transfer for low-resource languages that currently benefit less from manual prompting strategies.
Summary updated