Dynamic Visual Prompts That Know When to Back Off

A common way to adapt a large image-text model such as CLIP to a new task is prompt tuning: instead of changing the model's weights, we learn a few extra input vectors, called prompts, that steer the frozen model. Dynamic prompting goes one step further and generates a different prompt for each input image, which makes the adaptation far more expressive. The catch shows up under cross-dataset validation, where a model trained on one dataset (say, ImageNet) is tested without retraining on unrelated datasets (satellite images, textures, aircraft). The prompt generator was only ever trained on source images. When it meets a satellite photo, it still produces a confident, image-specific prompt, but that prompt is an extrapolation with nothing to justify it, and it can push the model's features away from what the pretrained model already knew.

We propose a dynamic visual prompt generator whose influence on each image is scaled by how typical that image looks relative to the source data. Typicality is measured with the frozen, unprompted encoder. Familiar-looking images get the full instance-specific adaptation. Unfamiliar images are smoothly returned to the frozen model's own representation. This works on a single test image, with no target-domain support batch and no target labels. The note is built around one falsifiable claim: dynamic prompts help cross-dataset transfer on source-like inputs and hurt it on atypical ones. A second question is whether a per-image back-off beats both static prompts with drift constraints and a fixed global mix of tuned and zero-shot predictions.

Core Claim

Dynamic prompting has so far entered cross-dataset evaluation mainly through the text side. Conditional Prompt Learning for Vision-Language Models (CoCoOp) adds an image-conditioned bias to text prompts through a small network. It is evaluated on ImageNet-to-10-dataset transfer, yet it still falls behind zero-shot CLIP on specialized targets like EuroSAT.

Inside the image encoder, the evidence runs the other way but covers only in-distribution tests. Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts (VAPT) generates layer-wise visual prompts from the input tokens. It argues that static prompt tokens are too limited to match input-dependent attention. It reports 60.1% on Stanford Dogs with 1% of the data where static visual prompt tuning collapses to 3.6%. However, it is evaluated only within datasets (VTAB-1K, FGVC).

The strongest cross-dataset results come from static prompts that are kept close to the pretrained representation. Prompt Tuning for CLIP on the Pretrained Manifold (ManiPT) shows that prompt-tuned features drift away from CLIP's original feature space, which it calls manifold drift. It counters this with cosine penalties and additive fusion with frozen features, and reaches 68.04% average accuracy on ImageNet-to-10-dataset transfer.

These three results are in tension. VAPT says adaptation must be input-dependent to be expressive. ManiPT says adaptation must be tightly anchored to survive transfer. Our claim is that both hold, but for different inputs. The anchor strength should itself depend on the input, set by a source-typicality score that the prompt cannot manipulate because it is computed by the frozen encoder.

Concretely, let $z_0(x)$ be the frozen CLIP image embedding of image $x$ and $h(x)$ the change that the dynamic visual prompts induce in the final embedding. The adapted feature is

$$f(x) = \mathrm{normalize}\big(z_0(x) + g(x)\, h(x)\big), \qquad g(x) = \sigma\!\left(\frac{\tau - d(x)}{T}\right),$$

where $d(x)$ is a distance from $z_0(x)$ to the source feature distribution. We use a Mahalanobis distance to class-conditional Gaussians fit on ImageNet training features. $\tau$ is set to a fixed percentile of $d$ on held-out source images, $T$ is a temperature, and $\sigma$ is the logistic function. When $g(x)=0$ the model is exactly zero-shot CLIP. When $g(x)=1$ it becomes ManiPT-style additive fusion, but with a dynamic rather than static prompt.

Research Program

The first stage is a diagnostic that fills a gap the literature leaves open. Nobody has tested in-backbone dynamic visual prompts under cross-dataset transfer. We train a VAPT-style generator on 16-shot ImageNet with a CLIP ViT-B/16 backbone. Its components are channel-wise convolutions, token-wise projectors and a shared projector MLP, trained with ManiPT's cosine consistency loss but without gating. We then evaluate it on the 10 standard target datasets and the four ImageNet shift variants.

Every target image is binned by $d(x)$. In each bin we measure the accuracy gain over zero-shot CLIP and the angular drift between $f(x)$ and $z_0(x)$. The prediction is that gain decreases, and drift increases, with typicality distance. If gain is flat or negative in every bin, dynamic visual prompting does not help cross-dataset transfer at all and gating is moot. We would report that as the finding.

This stratified analysis also addresses a known weakness of cross-dataset accuracy, which cannot separate feature misalignment from label-prior shift. We therefore report the per-bin analysis alongside class-balanced accuracy.

The second stage adds the gate. All gate parameters are fixed from source data only: the Gaussian fit, the choice of percentile for $\tau$, and $T$ chosen on an ImageNet held-out split. No target dataset or ImageNet variant is consulted. We also test two alternatives. The first uses a training-time surrogate that applies strong photometric and style augmentations to source images, together with a loss asking the gate-scaled prompt not to increase drift on them. The second is a learned gate. The fixed analytic gate is the default because it cannot overfit to source nuisance factors.

The third stage asks whether the effect is specific to the back-off. We separate the back-off from simply having less adaptation overall, and from knowing each dataset's best mixing weight in advance. These comparisons are described next.

Baselines

The naive reference point is zero-shot CLIP with the standard hand-written prompts. It is the "do nothing" option and the known winner on some specialized targets. The file gives its base-to-novel harmonic mean (71.70%, from CoCoOp) but no reported result found on Lacuna for its ImageNet-to-10-dataset average. We run it ourselves with the same prompt templates.

The strongest existing method on this exact protocol is ManiPT (Prompt Tuning for CLIP on the Pretrained Manifold). It reports 68.04% average cross-dataset transfer accuracy when trained on 16-shot ImageNet. It is the strongest competitor because it already contains our anchoring ingredient and differs only in using static prompts. We rerun it with our seeds, backbone and epoch budget, and also quote its published number. Beating it means a higher 10-dataset average than both our rerun and 68.04%, over three seeds. A per-dataset breakdown must show that the gain does not come from a single dataset.

We also include CoCoOp as the established dynamic-prompt method with cross-dataset results. The file lists no numeric cross-dataset figure for it, so we rerun it. We include Do-Prompt (Do-Prompt: Causal Interventions Meet Variational Prompt Bottlenecks), a static prompt with a capacity bottleneck and training-time interventions on a nuisance subspace, which reports a +9.20% ImageNet-to-EuroSAT gain over its base method. Finally, ungated VAPT is trained with the same cosine consistency loss. For the in-backbone variant we found no reported cross-dataset result on Lacuna, so this is new.

Two ablations isolate the key ingredient. The first is fixed-weight interpolation: $g(x)$ is replaced by a single constant chosen on the ImageNet held-out split, which tests whether per-image gating beats a global shrink. The second is per-dataset oracle interpolation: the constant is tuned on each target's test labels. This gives an upper bound for any dataset-level mixing, and matching it without labels would be strong evidence for the per-image signal. A shuffled-gate control randomly permutes $g(x)$ across images within each dataset. If the typicality score carries real information, shuffling should erase the gain.

All methods share the backbone, the 16-shot ImageNet data, the same number of epochs, and the same hyperparameter search budget on ImageNet validation only.

Why not just…?

Why not just use zero-shot CLIP on anything that looks unfamiliar? This amounts to a hard version of our gate, and it is a real alternative. We include a hard-threshold variant, with $T \to 0$, among the gate variants. Hard switching wastes adaptation on moderately shifted images such as ImageNet-Sketch or ImageNet-R, where CoCoOp and ManiPT both show tuned prompts beat zero-shot. Comparing soft and hard gates on those variants settles which is better.

Why not just use a static prompt with manifold constraints, as ManiPT does? This is the main threat. ManiPT already holds the best reported cross-dataset number, and static prompts are cheaper to batch: ManiPT notes CoCoOp trains 24x slower. Our answer is empirical. If gated dynamic prompts do not beat ManiPT on the 10-dataset average and on ImageNet variants, the extra expressiveness VAPT demonstrates in-distribution does not survive transfer. That result would be worth publishing on its own. We also report training and inference cost. The additive design inherits ManiPT's two-pass latency (6.96 vs ~3.5 ms/img), and our in-backbone generator adds little FLOP overhead, per VAPT's 0.6% figure over static visual prompt tuning.

Why not just average the predictions of the tuned and zero-shot models with one fixed weight? A global mix is the simplest way to trade plasticity for preservation, and it could plausibly capture most of the gain. The fixed-weight and oracle-weight ablations exist precisely to test this. Per-image gating is justified only if it beats the source-tuned fixed weight and approaches the oracle. If it does neither, we will conclude that the benefit is shrinkage, not instance awareness.

Why not adapt to the target domain with a small unlabeled batch, as VDPG does? Adapting to Distribution Shift by Visual Domain Prompt Generation generates domain prompts from a few unlabeled target images and reports 59.8% on DomainNet and 46.5% macro F1 on iWildCam. When such a batch exists this is likely stronger. We compare on DomainNet leave-one-domain-out and report how much of VDPG's gain a single-image method recovers. This matters for deployments where test images arrive one at a time from unknown sources.

Proposed Experiments

The main experiment trains on 16-shot ImageNet and evaluates zero-shot on Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT and UCF101. We report per-dataset and average top-1 accuracy over three seeds. Domain generalization is evaluated on ImageNet-V2, -Sketch, -A and -R. A second cross-dataset setting uses DomainNet leave-one-domain-out, with iWildCam as a sensor and location shift, and compares against VDPG's reported numbers.

The decisive checks come in a fixed order. First, in the stratified diagnostic, ungated dynamic prompts must show gain that declines with typicality distance, or the premise fails. Second, the gated model must beat ManiPT's 68.04% average and our ManiPT rerun, with the largest gains on EuroSAT, DTD and FGVCAircraft. It must also not lose ground on the ImageNet variants. Third, the gated model must beat the source-tuned fixed-weight interpolation, and the shuffled-gate control must lose the gain. Only that pattern supports the claim that per-image typicality, not generic shrinkage, carries the benefit.

A negative result at the first check would still yield the first cross-dataset measurement of in-backbone dynamic visual prompts. A negative result at the third would show that published cross-dataset gains from prompt anchoring are largely a mixing-weight effect.