Abstract
Prompt tuning, where a base pretrained model is adapted to downstream tasks via learned soft prompt vectors, has emerged as a promising approach for efficiently adapting large language models (LLMs). However, vanilla prompt tuning typically learns vectors from scratch for each task, failing to exploit cross-task knowledge. We propose Multitask Prompt Tuning (MPT), a framework that learns a single transferable prompt by distilling knowledge from multiple source tasks. By applying multiplicative low-rank updates to this shared prompt, MPT efficiently adapts to target tasks. Experiments across 23 NLP datasets show that MPT outperforms state-of-the-art methods and occasionally exceeds full finetuning performance while updating only 0.035% of the model's parameters.
Introduction
As Large Language Models (LLMs) scale to hundreds of billions of parameters, the traditional paradigm of Full Finetuning (FT)—updating every weight in the network for every new task—has become computationally and logistically prohibitive. Prompt Tuning (PT) offers a lightweight alternative by keeping the base model frozen and only optimizing a small sequence of continuous "soft prompt" vectors prepended to the input.
Despite its efficiency, vanilla PT faces two primary hurdles: the "Cold Start" problem, where prompts are initialized without prior task knowledge, and a persistent performance gap compared to full finetuning, particularly on smaller models or complex reasoning tasks. Multitask Prompt Tuning (MPT) addresses these challenges by enabling the sharing and transfer of knowledge across tasks within the prompt tuning framework.
The MPT Framework
MPT operates in two distinct phases: Source Training (Knowledge Distillation) and Target Adaptation.
1. Source Training and Decomposition
Rather than learning a monolithic prompt, MPT decomposes the prompt for each source task into a shared component and a task-specific component. The final prompt is the Hadamard product (element-wise multiplication) of a shared matrix ($P^*$) and a task-specific rank-one matrix (derived from vectors $u_k$ and $v_k$).
To ensure the shared matrix captures meaningful linguistic features, the authors employ Knowledge Distillation. Individual "Teacher" prompts are first trained for various source tasks. The MPT "Student" model is then trained to minimize the difference between its output distributions (and internal hidden states) and those of the teachers. This process distills the collective expertise of multiple source tasks into the shared matrix $P^*$.
2. Target Adaptation
When adapting to a new target task, the learned shared matrix $P^*$ serves as a powerful initialization. During this phase, MPT initializes new task-specific vectors while keeping the shared matrix available for further refinement. The framework utilizes a "two-speed" learning rate strategy, allowing the shared knowledge to be preserved while the task-specific components quickly specialize to the new domain.
Experimental Evaluation
MPT was evaluated on a comprehensive suite of 23 NLP datasets, using T5-Base and T5-Large as the underlying models. The evaluation covered several categories:
- General Language Understanding: GLUE and SuperGLUE benchmarks.
- Reading Comprehension: The MRQA benchmark.
- Natural Language Generation (NLG): E2E and WebNLG datasets.
- Cross-Domain Transfer: WinoGrande, Yelp-2, SciTail, and PAWS-Wiki.
Key Findings
Superior Parameter Efficiency
MPT achieves state-of-the-art performance while remaining exceptionally lean. It requires only approximately 77,000 tunable parameters per task—roughly 0.035% of the parameters required for full finetuning of a T5-Base model. Despite this, it outperformed previous multitask methods like ATTEMPT, which require significantly more task-specific parameters.
Performance Gains
On the challenging SuperGLUE benchmark, MPT demonstrated a 16.3% improvement over vanilla prompt tuning. In several instances, MPT's performance matched or exceeded that of full finetuning, suggesting that for many tasks, a well-conditioned soft prompt is as effective as updating the entire model architecture.
Cross-Task Synergy
One of the most striking results was the effectiveness of NLU-to-NLG transfer. Prompts distilled from "understanding" tasks (like classification and inference) significantly boosted performance on "generation" tasks. This indicates that the shared matrix $P^*$ captures fundamental language structures that are universal across different task formats.
Few-Shot Robustness
Because MPT begins with a pre-conditioned shared prompt, it excels in low-data regimes. In few-shot scenarios (4 to 32 examples), where vanilla PT often fails to converge or produces unstable results, MPT provides a stable and high-performing starting point.
Conclusion and Future Directions
Multitask Prompt Tuning represents a significant step toward making LLM adaptation both efficient and high-performing. By moving beyond learning prompts in isolation, MPT demonstrates that cross-task knowledge distillation can bridge the gap between lightweight tuning and full-scale finetuning.
While MPT shows a slight performance gap on complex Question Answering tasks (MRQA), its success across 23 datasets establishes it as a robust baseline for parameter-efficient transfer learning. Future research may explore automated source task selection and scaling the distillation process to hundreds of source tasks to further refine the shared knowledge base.
Summary updated