Abstract
Multitask Prompt Tuning (MPT) bridges the performance gap between lightweight prompt tuning and heavy full-model finetuning. By distilling knowledge from multiple source tasks into a single, decomposable shared prompt, MPT achieves state-of-the-art results across 23 NLP datasets while tuning only 0.035% of the parameters required for full finetuning.
The Challenge of Scaling LLM Adaptation
Adapting Large Language Models (LLMs) like T5 or GPT-3 to specific downstream tasks traditionally relies on Full Finetuning (FT)—updating every single parameter in the model. While highly effective, this approach presents a massive scaling challenge: deploying a model across 100 different tasks requires maintaining 100 separate copies of a billion-parameter model.
Prompt Tuning (PT) has emerged as a parameter-efficient alternative. Instead of altering the model's core weights, PT prepends a small set of learnable "soft prompt" vectors to the input. The base model remains completely frozen, and only the prompt is trained. However, PT has historically lagged behind FT in accuracy, is notoriously sensitive to initialization, and struggles to share general linguistic knowledge across different tasks.
The central question this research addresses is: Can we learn a single, universal prompt that captures cross-task knowledge to make adaptation both efficient and powerful?
Methods: The MPT Framework
To solve the limitations of standard prompt tuning, the authors introduce Multitask Prompt Tuning (MPT).
As illustrated above, MPT departs from existing aggregation methods by utilizing a two-stage framework centered on distillation and decomposition:
-
Source Training via Distillation: Instead of training a shared prompt directly on a massive multitask dataset (which often leads to "task interference"), MPT leverages Knowledge Distillation. The system first trains individual "teacher" prompts for various source tasks. Subsequently, a single "student" prompt is trained to mimic the behavior of these specialized teachers.
-
Prompt Decomposition: This is the core innovation of MPT. A standard prompt is represented as a matrix. MPT decomposes this into a shared matrix ($P^*$) and task-specific low-rank vectors ($u_k, v_k$). The actual prompt used for a task is the Hadamard product (element-wise multiplication) of the shared matrix and the rank-one matrix formed by the task vectors: $\hat{P}_k = P^* \circ (u_k \otimes v_k^T)$.
-
The "Slow" Weights ($P^*$): Capture general linguistic knowledge shared across all source tasks.
- The "Fast" Weights ($u_k, v_k$): Capture the specific nuances of a single task.
- Target Adaptation: When a new "target" task is introduced, it is initialized with the learned shared matrix and the averaged task vectors from the source phase. The model then performs a final round of highly efficient tuning on the target data.
Datasets and Evaluation
The researchers evaluated MPT across a rigorous suite of 23 NLP datasets:
- Source Tasks: 6 large-scale datasets (MNLI, QNLI, QQP, SST-2, SQuAD, ReCoRD) were used to build the initial shared prompt.
- Target Benchmarks:
- GLUE & SuperGLUE: Standard benchmarks for general language understanding.
- MRQA: Evaluated reading comprehension and question answering.
- NLG Tasks: E2E and WebNLG tested whether knowledge from understanding tasks transfers to text generation.
- Few-Shot: Evaluated on BoolQ, CB, and SciTail with extremely limited data (4 to 32 examples).
Key Insights and Results
- Unmatched Efficiency: As seen in the performance chart above, MPT occupies the ideal top-left quadrant. It uses only 0.035% of the parameters required for full finetuning (e.g., 77.6K vs. 220M for T5-Base) while maintaining highly competitive accuracy.
- State-of-the-Art Performance: On the SuperGLUE benchmark, MPT outperformed vanilla Prompt Tuning by 16.3% and surpassed the previous best multitask method (ATTEMPT), despite using significantly fewer task-specific parameters.
- Beating the Full Finetuning Baseline: In several instances, MPT actually outperformed full finetuning. This demonstrates that a well-initialized, frozen model can sometimes be more robust and generalize better than a model whose core weights have been entirely updated.
- Decomposition Prevents Interference: Simply sharing a prompt across tasks causes "interference"—learning Task A might overwrite knowledge useful for Task B. MPT's multiplicative low-rank update allows the model to "mask" or "highlight" different parts of the shared knowledge dynamically.
- Cross-Paradigm Transfer: Surprisingly, a prompt trained primarily on Natural Language Understanding (NLU) tasks significantly boosted performance on Natural Language Generation (NLG) tasks. This suggests the shared prompt learns fundamental properties of language rather than just task-specific shortcuts.
- Few-Shot Stability: In extreme low-data regimes (e.g., 4-shot learning), MPT proved significantly more stable and accurate than other parameter-efficient methods, which often fail to converge with minimal data.
Limitations and Future Work
While MPT represents a significant leap forward, some challenges remain. There is still a noticeable performance gap between MPT and full finetuning on highly complex Reading Comprehension tasks (like those in the MRQA benchmark).
Additionally, the current framework relies on a fixed set of 6 source tasks. Future research could explore automated methods for selecting the optimal source tasks to distill from for specific target domains. Finally, researchers noted that performance peaks at a prompt length of 300 tokens before overfitting occurs, leaving the optimal "memory capacity" for these prompts as an open area of investigation.
Summary updated