Visual Instance-aware Prompt Tuning

Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, Min Xu

Abstract: Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes. Instead, ViaPT overcomes these limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters compared to VPT-Deep. Extensive experiments across 34 diverse datasets demonstrate that our method consistently outperforms state-of-the-art baselines, establishing a new paradigm for analyzing and optimizing visual prompts for vision transformers.

Introduction

Adapting massive, pre-trained Vision Transformers (ViTs) to specific tasks often relies on Visual Prompt Tuning (VPT). Instead of updating the whole model, VPT adds a few learnable "tokens" (prompts) to the input. However, standard VPT uses dataset-level prompts: the same set of tokens is used for every single image in a dataset.

This strategy is often sub-optimal because a single set of prompts cannot capture the high variance found in complex datasets—such as the subtle differences between bird species or significant domain shifts in satellite imagery. Furthermore, existing VPT variants are effectively "corner cases": VPT-Shallow passes all prompt information forward without layer-specific adaptation, while VPT-Deep replaces prompts at every layer, potentially losing instance-specific context.

Methodology

ViaPT introduces two core innovations to move beyond "one-size-fits-all" prompting:

  1. Instance-aware Prompt Generator: Instead of static tokens, ViaPT uses a lightweight 2-layer convolutional encoder to analyze the input image and generate unique prompts. To ensure robustness, it employs a probabilistic approach where the generator predicts a mean ($\mu$) and standard deviation ($\sigma$), and the actual prompt is sampled from that distribution.
  2. Balanced Prompt Propagation (The PCA Trick): To bridge the gap between VPT-Shallow and VPT-Deep, ViaPT uses Principal Component Analysis (PCA). At each transformer layer, it takes the prompt features from the previous layer, reduces their dimensionality using PCA to retain the most "informative" components, and then pads the remaining dimensions with new, learnable parameters.
Overview of Visual Instance-aware Prompt Tuning (ViaPT) architecture, highlighting the instance-aware prompt generator (1) and the balanced prompt propagation via PCA (2).

Experimental Results

The researchers evaluated ViaPT on a massive suite of 34 diverse datasets across three major benchmarks: HTA (Heterogeneous Task Adaptation), VTAB-1k (Visual Task Adaptation Benchmark), and FGVC (Fine-Grained Visual Categorization).

  • State-of-the-Art Performance: ViaPT achieved a mean accuracy of 76.36% on VTAB-1k, outperforming strong baselines like SA²VP (75.83%) and VPT-Deep (69.43%).
  • Fine-Grained Mastery: On the FGVC benchmark, it reached 91.40%, notably excelling on difficult datasets like Stanford Dogs.
  • Parameter Efficiency: Despite the dynamic generation, ViaPT only tunes about 0.31% to 0.66% of the total backbone parameters, making it comparable in weight to much simpler methods.

Key Insights

One of the most elegant insights of this work is reframing existing VPT methods as endpoints on a spectrum of "information flow." By using PCA, the authors turned a binary choice (Shallow vs. Deep) into a tunable parameter $m$ that controls how much "memory" the prompts have.

Accuracy vs. reduced dimensionality m. ViaPT bridges the gap between VPT-Deep (m=0) and VPT-Shallow (m=768), finding an optimal balance through PCA.

As shown in the figure above, keeping all dimensions ($m=768$) results in VPT-Shallow, while keeping none ($m=0$) results in VPT-Deep. ViaPT finds a "sweet spot" (typically around $m=128$) that balances historical context with new adaptations. Unlike random dimension reduction, PCA acts as a semantic filter that forces the model to focus on the most task-relevant latent directions.

Conclusion

ViaPT demonstrates that instance sensitivity is crucial for high-performance visual prompt tuning. By generating image-specific prompts and balancing their propagation with PCA, the model produces sharper, more centralized attention on objects. While challenges remain in "Structured" tasks requiring precise 3D geometric reasoning, ViaPT establishes a new state-of-the-art for parameter-efficient fine-tuning in vision transformers.

Summary updated

Abstract

Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes. Instead, ViaPT overcomes these limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters compared to VPT-Deep. Extensive experiments across 34 diverse datasets demonstrate that our method consistently outperforms state-of-the-art baselines, establishing a new paradigm for analyzing and optimizing visual prompts for vision transformers.
BibTeX
@inproceedings{xiao2025visual,
  author = {Xi Xiao and Yunbei Zhang and Xingjian Li and Tianyang Wang and Xiao Wang and Yuxiang Wei and Jihun Hamm and Min Xu},
  title = {{Visual Instance-aware Prompt Tuning}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3754858},
}
Figure