Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris, Huan Sun, Yoon Kim

Abstract

Prompt tuning, where a base pretrained model is adapted to downstream tasks via learned soft prompt vectors, has emerged as a promising approach for efficiently adapting large language models (LLMs). However, vanilla prompt tuning typically learns vectors from scratch for each task, failing to exploit cross-task knowledge. We propose Multitask Prompt Tuning (MPT), a framework that learns a single transferable prompt by distilling knowledge from multiple source tasks. By applying multiplicative low-rank updates to this shared prompt, MPT efficiently adapts to target tasks. Experiments across 23 NLP datasets show that MPT outperforms state-of-the-art methods and occasionally exceeds full finetuning performance while updating only 0.035% of the model's parameters.

Introduction

As Large Language Models (LLMs) scale to hundreds of billions of parameters, the traditional paradigm of Full Finetuning (FT)—updating every weight in the network for every new task—has become computationally and logistically prohibitive. Prompt Tuning (PT) offers a lightweight alternative by keeping the base model frozen and only optimizing a small sequence of continuous "soft prompt" vectors prepended to the input.

Despite its efficiency, vanilla PT faces two primary hurdles: the "Cold Start" problem, where prompts are initialized without prior task knowledge, and a persistent performance gap compared to full finetuning, particularly on smaller models or complex reasoning tasks. Multitask Prompt Tuning (MPT) addresses these challenges by enabling the sharing and transfer of knowledge across tasks within the prompt tuning framework.

The MPT Framework

MPT operates in two distinct phases: Source Training (Knowledge Distillation) and Target Adaptation.

1. Source Training and Decomposition

Rather than learning a monolithic prompt, MPT decomposes the prompt for each source task into a shared component and a task-specific component. The final prompt is the Hadamard product (element-wise multiplication) of a shared matrix ($P^*$) and a task-specific rank-one matrix (derived from vectors $u_k$ and $v_k$).

To ensure the shared matrix captures meaningful linguistic features, the authors employ Knowledge Distillation. Individual "Teacher" prompts are first trained for various source tasks. The MPT "Student" model is then trained to minimize the difference between its output distributions (and internal hidden states) and those of the teachers. This process distills the collective expertise of multiple source tasks into the shared matrix $P^*$.

2. Target Adaptation

When adapting to a new target task, the learned shared matrix $P^*$ serves as a powerful initialization. During this phase, MPT initializes new task-specific vectors while keeping the shared matrix available for further refinement. The framework utilizes a "two-speed" learning rate strategy, allowing the shared knowledge to be preserved while the task-specific components quickly specialize to the new domain.

Experimental Evaluation

MPT was evaluated on a comprehensive suite of 23 NLP datasets, using T5-Base and T5-Large as the underlying models. The evaluation covered several categories:

  • General Language Understanding: GLUE and SuperGLUE benchmarks.
  • Reading Comprehension: The MRQA benchmark.
  • Natural Language Generation (NLG): E2E and WebNLG datasets.
  • Cross-Domain Transfer: WinoGrande, Yelp-2, SciTail, and PAWS-Wiki.

Key Findings

Superior Parameter Efficiency

MPT achieves state-of-the-art performance while remaining exceptionally lean. It requires only approximately 77,000 tunable parameters per task—roughly 0.035% of the parameters required for full finetuning of a T5-Base model. Despite this, it outperformed previous multitask methods like ATTEMPT, which require significantly more task-specific parameters.

Performance Gains

On the challenging SuperGLUE benchmark, MPT demonstrated a 16.3% improvement over vanilla prompt tuning. In several instances, MPT's performance matched or exceeded that of full finetuning, suggesting that for many tasks, a well-conditioned soft prompt is as effective as updating the entire model architecture.

Cross-Task Synergy

One of the most striking results was the effectiveness of NLU-to-NLG transfer. Prompts distilled from "understanding" tasks (like classification and inference) significantly boosted performance on "generation" tasks. This indicates that the shared matrix $P^*$ captures fundamental language structures that are universal across different task formats.

Few-Shot Robustness

Because MPT begins with a pre-conditioned shared prompt, it excels in low-data regimes. In few-shot scenarios (4 to 32 examples), where vanilla PT often fails to converge or produces unstable results, MPT provides a stable and high-performing starting point.

Conclusion and Future Directions

Multitask Prompt Tuning represents a significant step toward making LLM adaptation both efficient and high-performing. By moving beyond learning prompts in isolation, MPT demonstrates that cross-task knowledge distillation can bridge the gap between lightweight tuning and full-scale finetuning.

While MPT shows a slight performance gap on complex Question Answering tasks (MRQA), its success across 23 datasets establishes it as a robust baseline for parameter-efficient transfer learning. Future research may explore automated source task selection and scaling the distillation process to hundreds of source tasks to further refine the shared knowledge base.

Summary updated

Abstract

Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning 0.035% as many task-specific parameters.
BibTeX
@inproceedings{wang2023multitask,
  author = {Zhen Wang and Rameswar Panda and Leonid Karlinsky and Rogério Feris and Huan Sun and Yoon Kim},
  title = {{Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Nk2pDtuhTq},
}
Figure