Introduction
Fine-tuning massive pre-trained vision models (PVMs) has become computationally prohibitive for many downstream applications. Visual Prompt Tuning (VPT) has emerged as a parameter-efficient alternative, inserting a small number of learnable "prompt tokens" into the input space of Transformer blocks while keeping the backbone frozen.
However, existing VPT methods typically rely on a static distribution strategy: prompts are either placed only in the first layer (Shallow VPT) or distributed uniformly across all layers (Deep VPT). These "one-size-fits-all" approaches fail to account for the varying needs of different tasks. For instance, a task involving natural images might require more adaptation in early layers, whereas structured or synthetic tasks might benefit from deeper prompting. This rigidity leads to "idle" prompts in some layers and "prompt deficiency" in others, ultimately limiting performance.
The PRO-VPT Framework
To address these limitations, we propose PRO-VPT (iterative Prompt RelOcation-based VPT), which introduces Adaptive Distribution Optimization (ADO). Instead of a fixed layout, PRO-VPT dynamically relocates prompts during the training process to find a task-specific optimal distribution.
Nested Optimization Strategy
The framework operates via a nested optimization loop:
- Inner Loop (Prompt Tuning): Standard gradient descent updates the values of the prompt tokens to minimize task loss.
- Outer Loop (Distribution Adjustment): Periodically, the model evaluates the current distribution and relocates tokens from low-impact layers to high-impact ones.
Pruning and Allocation
The relocation process is divided into two distinct phases:
- Idleness Score-based Pruning: To avoid computationally expensive brute-force searches, we utilize a first-order Taylor expansion to calculate an "idleness score" for each prompt. This score estimates the prompt's contribution to loss reduction. Prompts with high idleness are pruned (released) from their current blocks.
- RL-based Allocation: The decision of where to re-allocate these released prompts is framed as a Reinforcement Learning (RL) problem. A lightweight policy network, trained via Proximal Policy Optimization (PPO), observes the model's state and selects the target block that would most benefit from additional prompting.
Experimental Evaluation
We evaluated PRO-VPT across several major benchmarks to test its versatility and performance.
Datasets and Benchmarks
- VTAB-1k: A suite of 19 diverse classification tasks (Natural, Specialized, and Structured).
- FGVC: Fine-Grained Visual Classification tasks, including CUB-200-2011 and Stanford Cars.
- Generalization: Object detection and semantic segmentation tasks on COCO and ADE20k.
Key Results
PRO-VPT achieves state-of-the-art (SOTA) results while maintaining high efficiency. On the VTAB-1k benchmark, it reached an average accuracy of 78.0%, outperforming previous leaders like iVPT (77.1%) and advanced adapter-based methods. On FGVC, it achieved 91.7% average accuracy, the highest among all prompt-based competitors.
Crucially, the computational overhead is minimal. The RL policy network adds only ~0.014M parameters, and the inference latency on VTAB-1k increases only slightly from 13.37 ms/img to 15.55 ms/img. Furthermore, relocation proves more stable than simply adding new prompts, as it preserves existing knowledge rather than introducing untrained parameters mid-stream.
Conclusion and Future Work
PRO-VPT demonstrates that where prompts are placed is just as important as their content. By treating prompt distribution as a dynamic, task-adaptive variable, we can significantly enhance the performance of parameter-efficient fine-tuning. Future research will explore extending this relocation logic to other PEFT methods, such as adaptively adjusting the rank distribution in Low-Rank Adaptation (LoRA).
Summary updated