Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding

Bolei Ma, Ercong Nie, Helmut Schmid, Hinrich Schütze

Abstract

While prompt-based finetuning has emerged as a superior paradigm for few-shot learning, its efficacy in full-data cross-lingual scenarios remains less understood. This paper investigates the performance of the PROFIT (PROmpt-based FInuetuning for cross-lingual Transfer) pipeline across various Multilingual Pretrained Language Models (MPLMs) and tasks. Our findings reveal that while prompting generally outperforms vanilla finetuning, its advantages are modulated by task complexity, model scale, and the linguistic proximity of target languages to the source.

Introduction

In the evolution of Multilingual Pretrained Language Models (MPLMs) such as mBERT and XLM-R, the standard approach for downstream task adaptation has been Vanilla Finetuning. This method typically involves appending a task-specific classifier head to the [CLS] token and training the model on labeled data. Recently, Prompt-Based Finetuning has challenged this convention. By reformulating tasks as "cloze-style" queries (e.g., converting a sentiment task into "This movie was [MASK]"), prompting allows models to leverage their original pretraining objectives more effectively.

While the benefits of prompting in data-scarce (few-shot) environments are well-documented, its performance in full-data regimes and its robustness during cross-lingual transfer—where a model is trained in one language and tested in others—have remained under-explored. This study evaluates whether prompting is a universal improvement or if it faces diminishing returns in large-scale multilingual contexts.

Methodology: The PROFIT Pipeline

The researchers introduce the PROFIT pipeline, which centers on the use of Pattern-Verbalizer Pairs (PVPs):

  1. Pattern: A template that transforms standard input into a prompt (e.g., "Input: [Sentence] Question: Is this a paraphrase? Answer: [MASK]").
  2. Verbalizer: A semantic mapping that translates discrete class labels into concrete vocabulary tokens (e.g., mapping Label 1 to "Yes" and Label 0 to "No").

By using actual words instead of arbitrary numerical labels, the model can tap into the semantic knowledge acquired during pretraining. The study compares PROFIT against Vanilla Finetuning and a zero-shot "Direct" baseline using mBERT and XLM-R. Experiments were conducted across a data spectrum ranging from $K=1$ (one example per class) to the full available dataset.

Experimental Design and Datasets

The evaluation spans three core Cross-Lingual Language Understanding (XLU) tasks:

  • Amazon Reviews: Multiclass sentiment analysis (1-5 stars) across 6 languages.
  • PAWS-X: Binary paraphrase identification across 7 languages.
  • XNLI: Natural Language Inference (entailment, neutral, contradiction) across 15 languages.

Key Insights

1. The Complexity Delay

The advantage of prompting is not uniform across task types. For simple tasks like sentiment analysis, the most significant gains occur at the $K=1$ level. However, for complex sentence-pair tasks like NLI or paraphrase detection, the performance gap between prompting and vanilla finetuning actually widens as more data is introduced, before eventually stabilizing. This suggests that for complex logical relationships, the model requires a baseline of examples to effectively "anchor" the logic of the prompt.

2. Model Saturation and Scale

Model architecture significantly influences the utility of prompting. The performance delta is notably larger for mBERT than for XLM-R. Because XLM-R is a larger model trained on more extensive data, its vanilla finetuning performance is already high, leaving less "headroom" for prompting to provide substantial improvements in full-data scenarios.

3. Linguistic Gravity and Pretraining

Prompting does not equalize performance across all languages. Cross-lingual transfer success remains heavily dependent on language similarity (proximity to English) and the pretraining corpus size for the target language. If a language was underrepresented during the model's initial pretraining phase, the semantic cues provided by a prompt are less effective at bridging the transfer gap.

Results and Performance

  • Full-Data Performance: PROFIT consistently outperformed Vanilla Finetuning. On average, PROFIT-mBERT yielded a 1.23% improvement, while PROFIT-XLM-R showed a more modest gain of 0.18%.
  • Few-Shot Dominance: The most dramatic results were observed in the $K=1$ sentiment analysis setting, where PROFIT-XLM-R achieved 33.63% accuracy, significantly outperforming the 21.71% achieved by Vanilla Finetuning.
  • Statistical Significance: T-tests confirmed that for mBERT, the improvements from prompting are statistically significant across all tasks. For XLM-R, the gains are highly significant in few-shot settings but become marginal as the dataset size increases.

Conclusion and Future Directions

The PROFIT pipeline demonstrates that prompt-based finetuning is a robust alternative to vanilla methods, particularly when leveraging the semantic meaning of labels. However, its "superpowers" are most evident in smaller models and data-constrained environments.

Future research should expand these insights into sequence labeling tasks (such as NER) and explore automated or "soft" prompt generation to further optimize cross-lingual transfer for low-resource languages that currently benefit less from manual prompting strategies.

Summary updated

Abstract

Multilingual pretrained language models (MPLMs) have demonstrated substantial performance improvements in zero-shot cross-lingual transfer across various natural language understanding tasks by finetuning MPLMs on task-specific labelled data of a source language (e.g. English) and evaluating on a wide range of target languages. Recent studies show that prompt-based finetuning surpasses regular finetuning in few-shot scenarios. However, the exploration of prompt-based learning in multilingual tasks remains limited. In this study, we propose the ProFiT pipeline to investigate the cross-lingual capabilities of Prompt-based Finetuning. We conduct comprehensive experiments on diverse cross-lingual language understanding tasks (sentiment classification, paraphrase identification, and natural language inference) and empirically analyze the variation trends of prompt-based finetuning performance in cross-lingual transfer across different few-shot and full-data settings. Our results reveal the effectiveness and versatility of prompt-based finetuning in cross-lingual language understanding. Our findings indicate that prompt-based finetuning outperforms vanilla finetuning in full-data scenarios and exhibits greater advantages in few-shot scenarios, with different performance patterns dependent on task types. Additionally, we analyze underlying factors such as language similarity and pretraining data size that impact the cross-lingual performance of prompt-based finetuning. Overall, our work provides valuable insights into the cross-lingual prowess of prompt-based finetuning.
BibTeX
@inproceedings{ma2023is,
  author = {Bolei Ma and Ercong Nie and Helmut Schmid and Hinrich Schütze},
  title = {{Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding}},
  booktitle = {KONVENS},
  year = {2023},
}
Figure