Image-Conditioned Dynamic Prompting for Vision-Language Models

This direction is merged into another direction: dynamic prompt generation (dynamic visual prompting). Its last version stays readable and citable here.

Version 1 of 156 papers69 conceptsfrozen 2026-10-04

In the standard paradigm of Vision-Language Models (VLMs) like CLIP, we typically use static text prompts to query an image. To find a dog, we might use the string "a photo of a dog." However, this "one-size-fits-all" approach ignores a fundamental reality of vision: the appearance of a "dog" changes drastically depending on the lighting, the breed, the camera angle, and the surrounding environment.

Researchers are moving toward Image-Conditioned Dynamic Prompting, a family of techniques that allow the linguistic representation of a category to adapt to the specific visual context of the image being processed. Instead of a fixed vector for "dog," the model generates a unique, instance-specific embedding that reflects how "dog-ness" manifests in that particular scene.

The Limitation of Static Semantics

The core problem with static prompts is the high intra-class variability found in the real world. As noted in OVFoodSeg by Xiongwei Wu and colleagues, a boiled egg and a scrambled egg are the same ingredient but look nothing alike. A static prompt for "egg" struggles to align with both.

Similarly, in domain generalization, a "road" looks different in a sunny photo than it does in a foggy one. Kibeom Hong and the authors of DPMFormer argue that rigid text prompts cause "semantic misalignment" when the visual domain shifts. If the text representation cannot "see" the fog, it cannot adjust its expectations for what a road should look like under those conditions.

How it Works: Bridging the Modalities

The intuition behind this research direction is to create a feedback loop where visual features inform the text encoder. This is generally achieved through three main mechanisms:

1. Context-Aware Refinement

Instead of passing a prompt through a text encoder and using the result as-is, models like DenseCLIP, developed by Yongming Rao and Guangyi Chen, use a transformer-based module to update text embeddings. By looking at the global visual context of the input image, the model "nudges" the text embedding toward the specific visual characteristics present in the scene.

2. Instance-Induced Prompt Generation

Some frameworks go further by generating the prompt itself from the image. In IIPAD, Wenxi Lv and Qinliang Su use a shared prompt generator that creates unique, instance-specific tokens for every image. This allows a single model to detect anomalies across diverse object categories by adapting its "definition of normal" to the specific texture and orientation of the object currently in view.

3. Hyper-Detailed Attribute Composition

For fine-grained tasks like pedestrian recognition or ship classification, a single word is often insufficient. Junyi Wu and the EVSITP team use image-conditioned dynamic prompts to capture nuances like style or lighting variations. By combining fixed templates with learnable, image-dependent prompts, the model "tailors" its linguistic understanding to the unique appearance of an attribute.

Why Dynamic Prompting Works

Dynamic prompting works because it reduces the search space for the model. In a zero-shot or open-vocabulary setting, the model has to match a visual region to one of thousands of possible concepts. By conditioning the text on the image, we effectively tell the text encoder: "We are looking at a maritime scene with high contrast; adjust the 'ship' embedding to prioritize these specific visual signatures."

This mechanism is particularly powerful in:

  • Semantic Segmentation: ZegCLIP by Ziqin Zhou shows that conditioning text-based queries on global image context helps the decoder resolve the relationship between general concepts and their specific local manifestations.
  • Anomaly Detection: Crane by Alireza Salehi uses context-guided prompt learning to incorporate visual hints about the object being analyzed, making the resulting embeddings more discriminative.
  • Domain Adaptation: DA-Ada by Haochen Li transforms static prompts into image-conditional embeddings, allowing detection heads to remain discriminative even as environmental conditions change.

Test-Time Adaptation and Inversion

A recent frontier in this area is performing this adaptation at test-time without any prior training on the target domain. InvSeg by Jiayi Lin introduces "prompt inversion," where the model optimizes text embeddings to match the specific visual context of an image during inference. This ensures that the linguistic concepts used for segmentation are perfectly aligned with the structural boundaries of the objects actually present in the image.

By treating the relationship between vision and language as a dynamic conversation rather than a static lookup table, these researchers are enabling VLMs to handle the messy, high-variance reality of the physical world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_image_conditioned_dynamic_prompting_for__39770c95,
  title = {Image-Conditioned Dynamic Prompting for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-conditioned-dynamic-prompting-for-vision-language-models/rdr_39770c95736643788f45474680c4bbde},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_image_conditioned_dynamic_prompting_for__39770c95_v1,
  title = {Image-Conditioned Dynamic Prompting for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-conditioned-dynamic-prompting-for-vision-language-models/rdr_39770c95736643788f45474680c4bbde/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}