Do We Really Need a Large Number of Visual Prompts?

Youngeun Kim, Yuhang Li, Abhishek Moitra, Priyadarshini Panda

Abstract

Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.

Introduction

As Vision Transformers (ViTs) grow larger, fine-tuning the entire network for every new task becomes prohibitively expensive. Visual Prompt Tuning (VPT) emerged as a parameter-efficient alternative, keeping the backbone frozen and only training a small set of "prompt tokens" prepended to the input. However, VPT introduces a hidden computational cost: because self-attention complexity scales quadratically with the number of tokens, adding many prompts (e.g., 200 for a ViT-B/16) can nearly double the computational overhead (FLOPs). This study investigates whether all these prompts are necessary and proposes a method to prune redundant prompts without sacrificing performance.

Theoretical Analysis and Prompt Condensation

The researchers first conducted a rank analysis of the self-attention matrix. Using mathematical propositions, they demonstrated that the rank—a proxy for expressive power—increases only logarithmically as more prompts are added. This implies a law of diminishing returns where additional prompts contribute progressively less to the model's capacity.

Normalized cumulative eigenvalues for different prompt configurations, illustrating the diminishing returns of adding more prompts.

To address this redundancy, the authors propose Prompt Condensation (PC), a three-step gradient-based pruning workflow:

  1. Importance Scoring: A first-order Taylor expansion estimates the impact of removing a specific prompt by multiplying the loss gradient by the prompt value.
  2. Global Selection: Prompts are ranked across the entire model rather than per layer, allowing the architecture to retain prompts in layers where they are most critical.
  3. Fine-tuning: A brief recovery fine-tuning session (approx. 20 epochs) is performed on the remaining subset of prompts.

Experimental Results

The approach was validated on the FGVC (Fine-Grained Visual Classification) and VTAB-1k benchmarks using ViT-B/16 and Swin-B backbones.

Accuracy vs. percentage of prompts for Stanford Cars, DMlab, and SmallNORB datasets.

Empirical results confirmed the "50% Rule": reducing the prompt count by half results in almost no accuracy loss across most datasets. Performance typically remains stable until the prompt count drops below a 40% threshold. The PC method successfully reduces the number of prompts by approximately 70% while maintaining baseline accuracy.

Beyond accuracy, the efficiency gains are significant. On a ViT-B/16, reducing prompts from 200 to 50 cuts computational overhead from 108.5% to just 26.1%. On hardware like the NVIDIA A100, this translates to a substantial reduction in inference latency (e.g., from 118ms to ~53ms for Stanford Cars).

Conclusion

This work demonstrates that the high computational cost of VPT can be mitigated through intelligent prompt pruning. By leveraging gradient-based importance scoring and global selection, Prompt Condensation achieves a ~70% reduction in prompt tokens with minimal accuracy impact. Future work may explore "train-from-scratch" sparse prompting to avoid the initial full-prompt training stage and dynamic task-specific condensation ratios.

Summary updated

Abstract

Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.
BibTeX
@article{kim2024do,
  author = {Youngeun Kim and Yuhang Li and Abhishek Moitra and Priyadarshini Panda},
  title = {{Do We Really Need a Large Number of Visual Prompts?}},
  journal = {Neural Networks},
  year = {2024},
  doi = {10.1016/j.neunet.2024.106390},
}
Figure