When adapting massive language models to downstream applications, full fine-tuning can be prohibitively expensive. Prompt tuning offers a parameter-efficient alternative by freezing the base model's weights and optimizing a small set of continuous vectors—called "soft prompts"—prepended to the input embeddings.
However, standard prompt tuning has a glaring weakness: when these soft prompts are initialized randomly, training is notoriously slow to converge and highly unstable, particularly in few-shot or low-resource scenarios.
To overcome this, a prominent research direction has emerged around cross-task prompt initialization (or prompt transfer). Instead of starting from scratch, researchers initialize a target task's soft prompt using vectors learned from related source tasks, intermediate tasks, or domain-specific pre-training. As surveyed by Li et al., this paradigm shifts prompt tuning from isolated optimization to a framework of continuous transfer learning, unlocking massive gains in both training speed and final accuracy.
The Intuition: Why Soft Prompts are Reusable
Why should a soft prompt trained on one task help a model solve an entirely different task? The answer lies in the geometric stability of frozen language models.
When you fine-tune an entire model, the internal representation space shifts continuously. But in prompt tuning, the base model's parameters are locked. This means the model's internal "landscape"—how it processes tokens, routes attention, and represents concepts—remains completely static. A soft prompt essentially acts as a set of virtual steering tokens that guide the input through this fixed landscape to trigger specific capabilities.
Because the underlying model geometry does not change, a soft prompt that successfully navigates the model for a source task (e.g., sentiment analysis) has already learned how to extract generalized linguistic structures, formatting rules, or reasoning paths from the frozen model. As demonstrated by Su et al., these learned vectors capture modular, transferable knowledge. Initializing a target task with a pre-trained prompt drops the optimization process directly into a highly relevant neighborhood of the parameter space, bypassing the chaotic early phases of training and accelerating convergence by up to 50x.
Random Initialization:
[Random Vectors] ──(Blind Search through Frozen LLM)──> Slow & Unstable Convergence
Cross-Task Transfer:
[Source Task Prompt] ──(Warm Start in Valid Region)───> Rapid & High-Ceiling Adaptation
Strategies for Selecting and Formatting Prompts
Simply grabbing a random prompt from another task is not enough; the transfer process must be strategic. Researchers have developed several sophisticated mechanisms to optimize how knowledge is transferred:
1. Retrieval and Affinity Matching
To avoid the expensive trial-and-error of testing multiple source tasks, Vu et al. introduced SPoT (Soft Prompt Transfer), which treats prompts as task embeddings. By measuring the similarity between task representations, they construct a prompt library to efficiently retrieve the most compatible source prompt for any new target task. Taking this further, Feng developed a learned affinity scoring function that predicts task transferability using these compact prompt embeddings, drastically reducing the computational overhead of source-task selection.
2. Averaging and Composition
When dealing with low-resource target domains, relying on a single source task can introduce bias. Bansal et al. demonstrated that initializing a target prompt by averaging the embeddings of multiple similar source prompts provides a more robust, generalized starting point. Similarly, Lin et al. proposed ACCEPT, which decomposes soft prompts into shared learnable "codewords" initialized from intermediate tasks, allowing the model to compose multi-task knowledge dynamically.
3. Structural Pruning
Not every token in a source prompt is beneficial to a target task; some can cause negative interference. To isolate the core transferable structure, Ma et al. developed XPrompt, a method that applies hierarchical structured pruning and weight rewinding to eliminate "negative" prompt tokens before transfer, ensuring only high-quality structural knowledge is passed forward.
Expanding Beyond Task-to-Task Transfer
The core philosophy of warm-starting optimization through learned continuous spaces has proven effective across several adjacent domains:
- Domain Adaptation: Instead of transferring from a specific supervised task, Bhambhoria et al. pre-trained soft "prefixes" on unlabeled domain-specific text (such as legal forums). This domain-level initialization allows the model to excel at specialized downstream tasks even when few-shot data is available.
- Black-Box Optimization: When model gradients are inaccessible (Language-Model-as-a-Service), users must optimize prompts via derivative-free queries. Sun et al. showed that initializing these black-box prompts with embeddings pre-trained on similar tasks significantly reduces the number of queries needed to reach peak performance. This concept of exploiting cross-model commonalities to bias a low-dimensional search space is also mirrored in query-efficient adversarial attacks, such as the framework designed by Zhang et al..
- Proxy-Model Initialization: Transfer can also occur across model scales. Xu et al. introduced GrokTransfer, which accelerates generalization ("grokking") in large target models by initializing their embeddings with weights learned from smaller, low-capacity proxy models that have already partially mapped the data's structural symmetries.
- Instruction-Tuned Foundations: On a macro scale, the benefits of structured initialization are evident when choosing base models. As detailed in the Flan Collection by Longpre et al., using an instruction-tuned model as a foundation for downstream fine-tuning acts as a massive behavioral warm-start, yielding faster convergence and higher accuracy ceilings than raw pre-trained bases.
Boundaries and Limitations
While prompt transfer is highly effective, it is not a silver bullet. The success of the transfer is tightly bound to the alignment between tasks. Jung et al. conducted an empirical study revealing that prompt transfer can occasionally fail or trigger catastrophic forgetting if there are severe mismatches in dataset scale, model size, or structural elements like answer formats.
Ultimately, cross-task prompt initialization works because it treats parameter-efficient tuning not as an isolated optimization problem, but as a process of navigating a shared, frozen latent landscape. By anchoring new tasks to regions of this landscape that have already been mapped by prior training, models can learn faster, perform better, and require drastically less data.
Key Foundational Work
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2022) — contributed 3 concept elements to this direction.
On Transferability of Prompt Tuning for Natural Language Processing (2022) — contributed 2 concept elements to this direction.
Few-shot Unified Question Answering: Tuning Models or Prompts? (2023) — contributed 2 concept elements to this direction.