Standard soft prompt tuning has emerged as a highly efficient paradigm for adapting large language models (LLMs) without modifying their underlying weights. By prepending a sequence of learnable continuous vectors to the input, we can steer a frozen model to perform specific downstream tasks.
However, traditional soft prompts are static: they learn a single, fixed set of vectors for an entire task. This creates a fundamental "one-size-fits-all" limitation. A static prompt must represent a global average strategy for a dataset, meaning it often fails to capture the unique linguistic nuances, domain shifts, or structural complexities of individual sentences.
To overcome this, a new paradigm has emerged: Input-Conditioned Dynamic Soft Prompting. Instead of treating a prompt as a static matrix of parameters, this approach parameterizes the prompt as a function of the input instance. By generating or selecting prompts on the fly, models can navigate the latent space of frozen networks with unprecedented precision.
The Core Intuition: Prompts as Functions
To understand why dynamic prompting works, consider how a transformer processes a prompt. A soft prompt acts as an "attention anchor." When the frozen model computes self-attention, the keys and values derived from the prompt vectors dictate how information flows from the input tokens to the final prediction.
In static prompt tuning, the prompt vectors $\mathbf{P}$ remain constant regardless of the input $x$:
$$\text{Key, Value} = \text{Linear}(\mathbf{P})$$
If a task contains highly diverse inputs—such as a mix of technical jargon and common language—a static prompt becomes a compromise that satisfies neither perfectly.
Dynamic soft prompting transforms the prompt into a conditional variable:
$$\mathbf{P}(x) = f_\theta(x)$$
Here, $f_\theta$ is a lightweight auxiliary network (such as a small transformer, a bottleneck MLP, or a recurrent module) that processes the input context and generates a customized soft prompt.
As Zhuofeng Wu et al. demonstrated with IDPG: An Instance-Dependent Prompt Generation Method, shifting from static task-level vectors to instance-dependent functions allows models to match the performance of heavy full-model fine-tuning while updating as little as 0.04% of the total parameters. The auxiliary network acts as a real-time translator, mapping the specific semantic characteristics of an input sentence directly into the optimal neighborhood of the frozen model's latent space.
Static Prompting:
[Static Task Prompt] + [Input A] ---> Frozen LLM ---> Output A
[Static Task Prompt] + [Input B] ---> Frozen LLM ---> Output B
Dynamic Prompting:
[Input A] ---> [Auxiliary Generator] ---> [Dynamic Prompt A] + [Input A] ---> Frozen LLM ---> Output A
[Input B] ---> [Auxiliary Generator] ---> [Dynamic Prompt B] + [Input B] ---> Frozen LLM ---> Output B
Key Mechanisms for Dynamic Generation
Researchers have explored several distinct architectural mechanisms to generate these input-dependent representations efficiently.
1. Lightweight Bottleneck Generators
To prevent the prompt generator from adding significant computational overhead, many frameworks use compressed bottleneck architectures. For example, Wei Zhu et al. introduced IAPT: Instance-Aware Prompt Tuning for Large Language Models, which uses a small bottleneck module to dynamically produce instruction-specific vectors from the hidden states of an input instruction. Because the generated prompts adapt so precisely to the instruction's semantic nuances, the authors found that they required as few as four soft tokens to outperform traditional static methods, preserving the model's valuable context window.
2. Discrete Codebook Stabilization
Generating continuous vectors completely from scratch can sometimes introduce training instability or lead to out-of-distribution generalization failures. To ground the generation process, Rishabh Bhardwaj et al. proposed Vector-Quantized Input-Contextualized Prompts (VIP) in Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding. VIP passes the input through a secondary encoder but stabilizes the output by mapping it to a discrete codebook of learned prompt vectors. This quantization step acts as a regularizer, ensuring that the generated prompts remain robust even when encountering out-of-domain data.
3. Late and Layer-Wise Conditioning
Standard prompt tuning prepends vectors exclusively at the initial embedding layer. However, as the signal travels deeper into the network, the influence of the prompt can degrade. To combat this, Tianxiang Sun et al. developed Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts. Instead of inserting prompts at the input, their method generates soft prompts dynamically based on the hidden state representations of intermediate layers. Shortening the signal path allows the dynamic prompt to interact more directly with high-level semantic features, doubling inference speed and halving memory consumption compared to deep static prompting.
Why It Works: Navigating Complex Latent Spaces
The power of dynamic soft prompting becomes most apparent in scenarios where static templates break down entirely.
- Probing and Knowledge Retrieval: Universal templates often fail to capture the specific linguistic context needed to extract factual knowledge from a model's weights. Dingcheng Li et al. showed in PromptGen: Automatically Generate Prompts using Generative Models that tailoring prompts to specific subject entities allows a sequence-to-sequence model to navigate the target model's latent space far more effectively, significantly boosting factual retrieval accuracy without leaking answers.
- Unlocking Memorization: Static prompts frequently underestimate how much training data an LLM has memorized. In Unlocking Memorization in Large Language Models with Dynamic Soft Prompting, Zhepeng Wang demonstrated that conditioning continuous prompt embeddings on specific prefix contexts increases the likelihood of triggering verbatim data extraction by up to 135%, proving that memorized patterns are highly context-dependent.
- Cross-Modal and Structural Bridging: Dynamic prompting also serves as an effective bridge between non-textual structures and language models. In Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting, the authors generated soft prompts dynamically from graph embeddings. This allowed structural graph logic to directly influence the frozen language model's attention mechanism, bypassing the need for slow, text-heavy descriptions of graph relations.
Alternative Paradigms: Generation vs. Selection
While generating continuous vectors via auxiliary networks is highly effective, a closely related sub-direction focuses on Dynamic Prompt Selection and Composition.
Instead of generating entirely new vectors, these methods maintain a pool or library of pre-trained soft tokens and retrieve or compose them based on the input. For instance, Srijan Bansal developed PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks, which uses an attention mechanism to dynamically calculate weights for various monolingual prompts based on the input sentence, aggregating them into a single composite prompt. Similarly, in audio-language modeling, the authors of LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs utilized a similarity-based retrieval mechanism to select soft tokens from a shared pool, balancing general auditory knowledge with task-specific requirements on the fly.
Whether through pure generation or intelligent selection, the underlying philosophy remains identical: the most effective way to steer a frozen model is to treat the prompt not as a static anchor, but as a dynamic reflection of the input itself.
Key Foundational Work
PromptGen: Automatically Generate Prompts using Generative Models (2022) — contributed 3 concept elements to this direction.
Prompting for a conversation: How to control a dialog model? (2022) — contributed 3 concept elements to this direction.
IDPG: An Instance-Dependent Prompt Generation Method (2022) — contributed 2 concept elements to this direction.