TL;DR
This paper investigates prompt tuning as an alternative to fine-tuning for cross-lingual natural language understanding (NLU) tasks. They demonstrate that prompt tuning, which tunes only a tiny fraction of parameters (0.1-0.3%), significantly outperforms traditional fine-tuning across various NLU tasks, achieving better cross-lingual transfer and more aligned decision boundaries.
What Problem Does This Paper Address?
Large pre-trained multilingual language models (e.g., mBERT, XLM-R) have shown impressive capabilities in zero-shot cross-lingual transfer, meaning they can perform tasks in languages they weren't explicitly trained on. However, the standard evaluation method, fine-tuning on English data and testing on other languages, has limitations. Fine-tuning is computationally expensive, requiring significant memory and storage for each task. Furthermore, it often results in a substantial performance gap between English and other languages, indicating suboptimal cross-lingual transfer. This paper aims to explore whether prompt tuning, a more parameter-efficient approach, can overcome these limitations and achieve superior cross-lingual performance.
Methods
The core of this paper's methodology is a comparison between fine-tuning and prompt tuning for cross-lingual NLU tasks.
- Fine-tuning: This is the traditional approach where the entire pre-trained multilingual language model is updated (tuned) on task-specific English data.
- Prompt Tuning: In this method, the pre-trained multilingual language model is kept frozen. Instead, a small set of "prompt" parameters (continuous vectors) are added and tuned. These prompts are prepended to the input embeddings, acting as task-specific instructions. The paper mentions using "prefix prompts" and "layer prompts," where prompts are integrated as past keys and values within each transformer layer.
The authors specifically investigate prompt tuning on various NLU tasks, including sentence classification, structure prediction, and question answering, using a large multilingual model (XLM-R).
Datasets
The experiments were conducted on four datasets from the XTREME benchmark:
- XNLI: Cross-lingual Natural Language Inference
- PAWS-X: Cross-lingual Adversarial Dataset for Paraphrase Identification
- UD-POS: Part-of-Speech Tagging on the Universal Dependencies
- XQuAD and TyDiQA-GoldP: Cross-lingual Question Answering
These datasets cover three categories of downstream tasks: sentence classification, structure prediction, and question answering.
Key Insights and Ideas
- Prompt Tuning Outperforms Fine-Tuning: The most striking insight is that prompt tuning, despite tuning a minuscule fraction of parameters, consistently achieves better zero-shot cross-lingual transfer performance than full fine-tuning. This challenges the conventional wisdom that fine-tuning is necessary for optimal performance.
- Reduced Cross-Lingual Gap: Prompt tuning significantly reduces the performance gap between English and other languages, suggesting it learns more generalizable representations.
- Aligned Decision Boundaries: Analysis using t-SNE visualizations reveals that prompt tuning leads to much better alignment of decision boundaries across different languages. This indicates that prompt tuning learns language-independent classifiers, which is crucial for effective cross-lingual transfer. Fine-tuning, in contrast, results in more divergent decision boundaries.
- Parameter Efficiency: The paper highlights the extreme parameter efficiency of prompt tuning, requiring only 0.1% to 0.3% of the original model's parameters to be tuned. This has significant implications for reducing computational costs and storage requirements.
- Robustness: Prompt tuning makes fewer drastic changes to the original model's representations compared to fine-tuning, leading to more robust transfer and potentially less overfitting.
Key Results
- Performance Gains: Table 1 shows that prompt tuning consistently outperforms fine-tuning across sentence classification, structured prediction, and question answering tasks. For instance, on XQUAD, prompt tuning achieves 79.0/64.1 (F1/EM) compared to fine-tuning's 77.2/61.6.
- Reduced Transfer Gap: Table 2 quantifies the cross-lingual transfer gap (difference between English performance and average performance on other languages). Prompt tuning reduces this gap significantly, from 10.2% to 9.7% for XNLI and 24.3% to 20.7% for XQUAD, indicating better generalization.
- Parameter Efficiency: The paper emphasizes that prompt tuning uses only 0.1% to 0.3% of additional parameters compared to the original model.
- Variance Reduction: Prompt tuning also exhibits smaller variance in performance across tasks and languages compared to fine-tuning.
Limitations and Future Work
- Computational Resources: The authors acknowledge that conducting these experiments, especially on large multilingual models, requires substantial GPU resources (A100s with 40G memory).
- Model Size: While prompt tuning shows promise, the paper notes that for smaller models (e.g., mBERT), fine-tuning might still perform slightly better on English, though prompt tuning matches fine-tuning on other languages. This suggests that model size remains a factor.
- Future Work: The authors suggest exploring encoder-decoder models and other task types beyond NLU. They also hint at potential for further improvements with more advanced prompt tuning methods, especially for smaller models. The current work focuses on encoder-only models for NLU tasks.
Summary updated