Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More

Fred Philippy, Siwen Guo, Shohreh Haddadan, Cedric Lothritz, Jacques Klein, Tegawendé F. BissyandéMOOMIN (co-located With EACL ) 2024 · OpenReview

TL;DR

This paper shows that keeping pre-trained language models (PLMs) frozen while only training soft prompts (SPT) significantly improves cross-lingual transfer performance, especially for linguistically distant languages. This "less is more" approach not only drastically reduces computational costs but also enhances model robustness and efficiency.

What Problem Does This Paper Address?

Adapting large pre-trained language models (PLMs) to new tasks, especially in different languages, is computationally expensive and can lead to "catastrophic forgetting" of the model's original knowledge. For cross-lingual transfer, where knowledge is moved from a high-resource language (like English) to a low-resource one, existing methods often involve fine-tuning the entire model, which exacerbates these issues. This can result in poor performance, particularly when the target language is linguistically distant from the source. The paper aims to address this by investigating Soft Prompt Tuning (SPT), a parameter-efficient method, and demonstrating its effectiveness for cross-lingual transfer while maintaining model freezing.

Methods

The core of this paper's methodology revolves around Soft Prompt Tuning (SPT). SPT involves inserting a small set of learnable embeddings, called "soft prompts," into the input layer of a PLM. Crucially, the authors adhere to the original SPT approach by keeping all the PLM's parameters frozen and only training these soft prompts. This contrasts with some prior works that fine-tuned the entire model or added new classifiers.

The paper explores two key aspects of SPT for cross-lingual transfer:

  • Model Freezing: The central hypothesis is that freezing the PLM's parameters, while only training the soft prompt, leads to better cross-lingual transfer.
  • Prompt Engineering: They investigate how prompt length and reparameterization techniques affect cross-lingual transfer performance. Reparameterization involves using additional neural network layers (like an MLP) to process the prompt embeddings before they are fed into the PLM.

Datasets

The primary dataset used for evaluation is SIB-200. This is a topic classification dataset that stands out for its extensive coverage, encompassing 200 languages and dialects. This broad scope allows for more comprehensive observations on cross-lingual transfer compared to previous studies that often focused on smaller language sets. The task is to classify text samples into one of seven categories: science/technology, travel, politics, sports, health, entertainment, and geography.

Key Insights and Ideas

  • Freezing is Key for Cross-Lingual Transfer: The most significant finding is that keeping the PLM parameters frozen enhances cross-lingual transfer performance, particularly for languages that are linguistically distant from the source language. This challenges the intuition that more fine-tuning always leads to better results.
  • Parameter Efficiency Pays Off: By only training a small number of soft prompt parameters (less than 0.01% of the total model parameters), the method achieves comparable or even superior performance to full-model fine-tuning, while drastically reducing computational costs and storage requirements.
  • Prompt Length Matters: There's an optimal prompt length. Making prompts too long can lead to a degradation in cross-lingual transfer performance.
  • Reparameterization is Language-Dependent: The decision to use prompt reparameterization is not universally beneficial. Some languages (e.g., certain Indo-Aryan and Dravidian languages) benefit significantly, while others (e.g., Atlantic-Congo languages) see performance declines. This suggests a need for tailored approaches based on language families or specific linguistic characteristics.
  • Bridging the Transfer Gap: Freezing the model appears to mitigate the bias towards linguistically similar languages, helping to close the performance gap between similar and distant language pairs.

Key Results

The paper presents several quantitative and qualitative results:

  • Improved Accuracy with Freezing: Figure 2 shows a consistent improvement in average cross-lingual transfer accuracy across all tested models (BLOOM and XGLM) when model parameters are frozen (w/ MF) compared to when they are not (w/o MF). For instance, BLOOM-560M sees an increase in accuracy when frozen.
  • Correlation with Linguistic Similarity: Table 1 indicates that the correlation between transfer performance and linguistic similarity metrics (like syntactic, geographic, genetic distance) decreases when the model is frozen. This supports the idea that freezing makes transfer less biased towards similar languages.
  • Prompt Length Impact: Figure 4 demonstrates that cross-lingual transfer performance degrades when prompt lengths exceed a certain optimal point (e.g., lengths of 10 or 20 tokens are generally better than 30).
  • Reparameterization Effects: Figure 5 (Appendix C) shows significant performance variations due to reparameterization across different languages. For BLOOM, some languages experience drops of over 24-31% (e.g., Yoruba, Twi), while others see gains of up to 29% (e.g., Urdu, Hindi). XGLM shows more balanced results, with some Latin-script or Dravidian languages benefiting.
  • Computational Savings: The paper highlights that fine-tuning only the soft prompt requires less than 100KB of storage, compared to gigabytes for the full model. Training time is also significantly reduced, being less than half that of full-model training.

Limitations and Future Work

The authors acknowledge several limitations:

  • Instability of Few-Shot Fine-Tuning: The inherent instability of few-shot learning can affect the robustness of evaluations. While they mitigated this by running experiments multiple times with different random seeds, further research is needed.
  • Performance Drop for Similar Languages: While the method enhances transfer to distant languages, it sometimes lowers performance for linguistically similar ones. Future work could aim for more balanced performance across all language types.
  • Tailoring Reparameterization: The decision to use prompt reparameterization should not be uniform. Future research should focus on tailoring these techniques based on specific target languages or language families.

The paper suggests that future work should continue to explore how non-linguistic factors like prompt design can be optimized for cross-lingual transfer, especially as models continue to scale to billions or trillions of parameters, making parameter efficiency even more critical.

Summary updated

Abstract

Soft Prompt Tuning (SPT) is a parameter-efficient method for adapting pre-trained language models (PLMs) to specific tasks by inserting learnable embeddings, or soft prompts, at the input layer of the PLM, without modifying its parameters. This paper investigates the potential of SPT for cross-lingual transfer. Unlike previous studies on SPT for cross-lingual transfer that often fine-tune both the soft prompt and the model parameters, we adhere to the original intent of SPT by keeping the model parameters frozen and only training the soft prompt. This does not only reduce the computational cost and storage overhead of full-model fine-tuning, but we also demonstrate that this very parameter efficiency intrinsic to SPT can enhance cross-lingual transfer performance to linguistically distant languages. Moreover, we explore how different factors related to the prompt, such as the length or its reparameterization, affect cross-lingual transfer performance.
BibTeX
@inproceedings{philippy2024soft,
  author = {Fred Philippy and Siwen Guo and Shohreh Haddadan and Cedric Lothritz and Jacques Klein and Tegawendé F. Bissyandé},
  title = {{Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More}},
  booktitle = {MOOMIN (co-located With EACL )},
  year = {2024},
  url = {https://openreview.net/forum?id=VEineMBjs2},
}
Figure