When large-scale vision-language models (VLMs) like CLIP first arrived, the primary way to adapt them to specific tasks was through manual prompt engineering—carefully crafting strings like "a photo of a [CLASS]." Researchers soon realized that these prompts could be learned as continuous vectors, a technique known as Context Optimization (CoOp). However, a significant problem emerged: these learned prompts were static. They remained identical for every image in a dataset, causing the model to overfit to specific training categories and struggle with unseen "novel" classes.
The research direction of Dynamic Prompting (or Conditional Prompting) solves this by transforming prompts from fixed parameters into input-dependent variables. Instead of a single "one-size-fits-all" instruction, the model uses a lightweight auxiliary network to generate a unique prompt for every individual image.
The Limits of Static Instructions
The shift toward dynamic prompting was catalyzed by the observation that static prompts are inherently rigid. In Conditional Prompt Learning for Vision-Language Models, Kaiyang Zhou and colleagues demonstrated that while static prompts perform well on seen data, they fail to generalize because they don't account for the visual variance within a class.
If a model learns a static prompt for "dog" based only on golden retrievers, it may fail when it encounters a husky. By making the prompt conditional on the input image—a method called CoCoOp—the model can adjust its internal "focus" based on the specific visual features of the instance at hand, preserving the model's open-world flexibility.
From Fixed Vectors to Dynamic Generators
The core mechanism of this research area is the Prompt Generation Network (PGN) or "Meta-Network." Rather than storing prompt tokens in a lookup table, these systems use a small neural network to map an image's global features into the prompt space.
As Jochem Loedeman and Yuki M. Asano describe in Prompt Generation Networks for Efficient Adaptation of Frozen Vision Transformers, this approach allows a frozen backbone to be steered with extreme precision. By using a "Token Library" and a lightweight backbone, PGNs can match the performance of full fine-tuning while updating 100x fewer parameters. This is particularly effective in "black-box" scenarios where the underlying model weights are inaccessible via an API, as explored by Changdae Oh in BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning.
Steering Attention with Instance Context
Why does this work so much better than static prompting? The intuition lies in how transformers process information. A prompt acts as a set of "virtual" tokens that influence the attention mechanism of the entire model.
In DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for Medical Image Analysis, Along He shows that dynamic prompts act as queries that interact with image features via cross-attention. This allows the adaptation tokens to "pick up" sample-specific knowledge—such as the unique texture of a lesion or the specific lighting of a medical scan—before the main transformer layers process the data.
This instance-level personalization is taken further in Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning, where Huy Nguyen and Nhat Ho reinterpret prompts as "adaptive experts." By making prompts respond to input features at every layer, they achieved a 56% accuracy boost in low-data settings compared to standard static visual prompting.
Generalization through Content Awareness
Dynamic prompting is especially powerful for Domain Generalization—the ability to work on data that looks different from the training set (e.g., moving from photos to sketches).
Methods like ProS (Prompting-to-Simulate) by Jingkuan Song use a simulator to generate content-aware prompts for unseen scenarios. Similarly, Min Zhang introduced CoDoL, which uses a bottleneck network to capture instance-specific details that a generic "domain average" would miss.
By grounding the prompt in the actual content of the image—rather than just the class label—these models avoid the "semantic shift" that usually degrades performance when a model encounters a new environment. As Runpeng Yu demonstrates in Attention Prompting on Image for Large Vision-Language Models, this can even be used to generate query-dependent heatmaps that highlight relevant regions of an image, effectively telling the model not just what to look for, but where to look based on the specific question asked.