Visual Prompt Tuning for Frozen Foundation Models

67 concepts59 papers1 proposal

When large vision models are trained on massive datasets, they develop a rich, general-purpose understanding of visual structures. However, adapting these models to specific downstream tasks—such as medical imaging, object tracking, or semantic segmentation—traditionally requires full fine-tuning. Full fine-tuning updates all model weights, which is computationally expensive and risks destroying the model's foundational capabilities through catastrophic forgetting.

As highlighted by Cheng Han et al. in Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?, full fine-tuning can distort valuable pre-trained features, whereas parameter-efficient alternatives can achieve superior performance by keeping the backbone frozen. This realization has driven the rise of Visual Prompt Tuning (VPT): an adaptation paradigm that leaves the core model weights completely untouched, instead optimizing a tiny fraction of parameters introduced directly into the input or sequence space.

Input-Space Prompting: Reprogramming the Pixels

One intuitive way to guide a frozen model is to modify the input data itself. Instead of changing how the model thinks, we change what it sees. In input-space prompt tuning, a small set of learnable parameters is added directly to the input image pixels, often as a border, a localized patch, or a subtle pixel-level perturbation.

As Junyang Wu et al. demonstrate in Unleashing the Power of Visual Prompting At the Pixel Level, optimizing these pixel-level perturbations can outperform traditional linear probing while requiring significantly fewer tunable parameters.

Why does this work? The learnable pixels act as a specialized input transformation or "lens filter." They mathematically shift the distribution of the downstream dataset into the feature space that the frozen model already understands. This approach is highly effective for deploying models to resource-constrained edge devices, as explored by Jiaming Liu et al. in Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-World.

Furthermore, because input-space prompting only modifies the input image, it enables the adaptation of black-box models where internal weights and gradients are completely inaccessible. For instance, Jing Zhang leverages this advantage in LaViP: Language-Grounded Visual Prompting to optimize vision-language models efficiently without modifying internal layers.

Sequence-Space Prompting: Steering the Attention

For models based on the Vision Transformer (ViT) architecture, adaptation can also occur within the sequence space. Instead of altering raw pixels, researchers inject learnable token vectors into the input sequence alongside the standard patch embeddings.

In this setup, the transformer backbone remains frozen, but the added prompt tokens are fully optimized during training. As Li Ren et al. explain in Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning, these prompt vectors interact directly with the image patches through the model's self-attention layers. This allows the prompts to dynamically steer the internal representations and attention maps toward task-specific features without altering the underlying projection weights.

The parameter efficiency of this method can be extreme. Tristan Piater et al. demonstrate in Prompt-Tuning SAM: From Generalist to Specialist with only 2048 Parameters and 16 Training Images that the Segment Anything Model (SAM) can be adapted to specialized medical and microscopic domains using just 2,048 tunable parameters. Rather than retraining the massive transformer blocks, the learned tokens simply instruct the existing attention mechanism on which boundaries to prioritize.

Why Visual Prompts Work: The Geometry of Steering

To understand why visual prompting is so effective, it helps to view a frozen foundation model as a fixed, highly complex feature manifold. Large-scale pre-training maps the visual world into a structured space where geometric relationships correspond to semantic concepts.

When adapting to a new task, we rarely need to construct an entirely new manifold. Instead, we need to find the right pathway or projection within the existing space. Prompts act as a low-dimensional steering mechanism. By introducing a small number of tunable parameters, prompting shifts the context of the input data, allowing the model to activate latent capabilities that are already present but dormant.

This explains why visual prompting excels at handling extreme environmental shifts. For example, Jiawen Zhu et al. in DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVs adapt a frozen daytime object tracker to nighttime conditions by injecting lightweight "darkness clue" prompts. This preserves the model's core tracking logic while contextualizing it for low-light environments.

Similarly, Chang'an Yi et al. leverage input-space prompts in A Critical Look at Classic Test-Time Adaptation Methods in Semantic Segmentation to efficiently adapt segmentation models to new test distributions on the fly.

Ultimately, visual prompt tuning demonstrates that large vision models do not always need to be rebuilt or heavily modified to learn new domains. Often, providing the right contextual nudge in the input or sequence space is all it takes to unlock their full potential.

Key Foundational Work

Visual Prompt Tuning for Generative Transfer Learning (2023) — contributed 3 concept elements to this direction.
Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting (2023) — contributed 2 concept elements to this direction.
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visual_prompt_tuning_eabe19fb,
  title = {visual prompt tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-prompt-tuning/rdr_eabe19fb7fcb418ebb1a2e115688037d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_visual_prompt_tuning_eabe19fb_v1,
  title = {visual prompt tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-prompt-tuning/rdr_eabe19fb7fcb418ebb1a2e115688037d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake