Transferable Soft Prompts for Efficient Model Adaptation

This direction is split into other directions: cross-task prompt initialization. Its last version stays readable and citable here.

Version 1 of 128 papers39 conceptsfrozen 2026-10-04

Prompt tuning has emerged as a powerful, parameter-efficient alternative to full-model fine-tuning. Instead of updating billions of parameters, we prepend a small set of learnable "soft prompt" vectors to the input, keeping the underlying language model frozen. However, prompt tuning faces a significant hurdle: it is notoriously slow to converge and often struggles in few-shot scenarios where data is scarce.

Recent research has uncovered a solution to this "cold start" problem. It turns out that soft prompts are not just task-specific hacks; they are modular units of knowledge that can be transferred, reused, and even pre-trained across different tasks and architectures.

The Modular Nature of Soft Prompts

In standard prompt tuning, we initialize soft prompts randomly. This forces the optimization process to find a path through the model's high-dimensional space from scratch. Yusheng Su, Maosong Sun, and their colleagues demonstrated in On Transferability of Prompt Tuning for Natural Language Processing that this is unnecessary. By initializing a target task's prompt with vectors learned from a related source task, researchers achieved up to a 50x speedup in convergence.

This works because soft prompts capture "how to perform a task" in a format the frozen model already understands. This modularity allows a prompt learned on one sentiment analysis dataset to jumpstart learning on another, even if the domain or the model architecture changes.

Finding the Right Starting Point

The success of transfer depends heavily on the relationship between the source and target tasks. If the tasks are too different, the transferred prompt might provide a "negative" initialization. To solve this, researchers have developed several strategies for selecting or combining source knowledge:

  • Similarity-Based Retrieval: Yuhui Zuo and Guoyong Cai proposed calculating the similarity between a new input and a library of previously learned prompts. This allows a system to "borrow" the most relevant knowledge for zero-shot or few-shot adaptation.
  • Prompt Averaging: For unified question answering, Srijan Bansal and Semih Yavuz found that averaging the embeddings of prompts from similar source tasks provides a robust starting point that outperforms random initialization in low-resource settings.
  • Learned Affinity: Lingyun Feng introduced a learned affinity scoring function that treats soft prompts as compact task embeddings to predict transferability without expensive brute-force testing.

Pre-training the Prompt

If prompts are transferable, can we pre-train them like we pre-train models? Sheng Shen and Joseph E. Gonzalez explored this in Multitask Vision-Language Prompt Tuning. By training a shared prompt vector across a diverse set of source tasks, they created a "foundation prompt" that captures broad cross-domain concepts. This generalized initialization significantly boosts performance on downstream target tasks, suggesting that prompts benefit from their own dedicated pre-training phase.

Transfer in Restricted Environments

The transferability of prompts is particularly valuable when we cannot access the model's internal gradients—a scenario known as "Black-Box Tuning." Tianxiang Sun and Xipeng Qiu showed that the effectiveness of derivative-free optimization is greatly increased when using prompt embeddings pre-trained on similar tasks. By leveraging the low "intrinsic dimensionality" of the model, these transferred prompts allow for high-performance adaptation through simple API queries.

Even in the legal domain, where tasks are highly specialized, the principle holds. Joel Niklaus and Peter Henderson demonstrated that starting from a model already tuned for general instruction-following provides a superior foundation for learning complex legal reasoning, as the ability to follow instructions is itself a transferable skill.

Why It Works: The Intuition

The reason soft prompt transfer works is that pre-trained language models possess a latent "task space." Random initialization starts the search at a random point in this space, often far from the manifold of useful instructions. Transfer learning provides a "warm start" near a region of the space that the model already recognizes as a valid task structure.

Whether it's through hierarchical pruning to remove "negative" tokens as suggested by Fang Ma, or scaling prompts to improve few-shot generation as shown by Ting Hu, the core insight remains: the knowledge required to adapt a model is modular, and once captured in a prompt, it can be moved across the landscape of AI tasks.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_transferable_soft_prompts_for_efficient__2625db33,
  title = {Transferable Soft Prompts for Efficient Model Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transferable-soft-prompts-for-efficient-model-adaptation/rdr_2625db33927c477398d412e94eb2b849},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_transferable_soft_prompts_for_efficient__2625db33_v1,
  title = {Transferable Soft Prompts for Efficient Model Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transferable-soft-prompts-for-efficient-model-adaptation/rdr_2625db33927c477398d412e94eb2b849/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}