Iterative Prompt Relocation for Distribution-Adaptive Visual Prompt Tuning

Chikai Shang, Mengke Li, Yiqun Zhang, Zhen Chen, Jinlin Wu, Fangqing Gu, Yang Lu, Yiu-ming CheungarXiv 2025 · DOI · arXiv

Introduction

Fine-tuning massive pre-trained vision models (PVMs) has become computationally prohibitive for many downstream applications. Visual Prompt Tuning (VPT) has emerged as a parameter-efficient alternative, inserting a small number of learnable "prompt tokens" into the input space of Transformer blocks while keeping the backbone frozen.

However, existing VPT methods typically rely on a static distribution strategy: prompts are either placed only in the first layer (Shallow VPT) or distributed uniformly across all layers (Deep VPT). These "one-size-fits-all" approaches fail to account for the varying needs of different tasks. For instance, a task involving natural images might require more adaptation in early layers, whereas structured or synthetic tasks might benefit from deeper prompting. This rigidity leads to "idle" prompts in some layers and "prompt deficiency" in others, ultimately limiting performance.

Comparison between fixed prompt distributions in prior art and the proposed adaptive distribution.

The PRO-VPT Framework

To address these limitations, we propose PRO-VPT (iterative Prompt RelOcation-based VPT), which introduces Adaptive Distribution Optimization (ADO). Instead of a fixed layout, PRO-VPT dynamically relocates prompts during the training process to find a task-specific optimal distribution.

Nested Optimization Strategy

The framework operates via a nested optimization loop:

  1. Inner Loop (Prompt Tuning): Standard gradient descent updates the values of the prompt tokens to minimize task loss.
  2. Outer Loop (Distribution Adjustment): Periodically, the model evaluates the current distribution and relocates tokens from low-impact layers to high-impact ones.

Pruning and Allocation

The relocation process is divided into two distinct phases:

  • Idleness Score-based Pruning: To avoid computationally expensive brute-force searches, we utilize a first-order Taylor expansion to calculate an "idleness score" for each prompt. This score estimates the prompt's contribution to loss reduction. Prompts with high idleness are pruned (released) from their current blocks.
  • RL-based Allocation: The decision of where to re-allocate these released prompts is framed as a Reinforcement Learning (RL) problem. A lightweight policy network, trained via Proximal Policy Optimization (PPO), observes the model's state and selects the target block that would most benefit from additional prompting.
The PRO-VPT framework featuring nested optimization with idleness-based pruning and RL-based allocation.

Experimental Evaluation

We evaluated PRO-VPT across several major benchmarks to test its versatility and performance.

Datasets and Benchmarks

  • VTAB-1k: A suite of 19 diverse classification tasks (Natural, Specialized, and Structured).
  • FGVC: Fine-Grained Visual Classification tasks, including CUB-200-2011 and Stanford Cars.
  • Generalization: Object detection and semantic segmentation tasks on COCO and ADE20k.

Key Results

PRO-VPT achieves state-of-the-art (SOTA) results while maintaining high efficiency. On the VTAB-1k benchmark, it reached an average accuracy of 78.0%, outperforming previous leaders like iVPT (77.1%) and advanced adapter-based methods. On FGVC, it achieved 91.7% average accuracy, the highest among all prompt-based competitors.

Crucially, the computational overhead is minimal. The RL policy network adds only ~0.014M parameters, and the inference latency on VTAB-1k increases only slightly from 13.37 ms/img to 15.55 ms/img. Furthermore, relocation proves more stable than simply adding new prompts, as it preserves existing knowledge rather than introducing untrained parameters mid-stream.

Conclusion and Future Work

PRO-VPT demonstrates that where prompts are placed is just as important as their content. By treating prompt distribution as a dynamic, task-adaptive variable, we can significantly enhance the performance of parameter-efficient fine-tuning. Future research will explore extending this relocation logic to other PEFT methods, such as adaptively adjusting the rank distribution in Low-Rank Adaptation (LoRA).

Summary updated

Abstract

Visual prompt tuning (VPT), i.e., fine-tuning some lightweight prompt tokens, provides an efficient and effective approach for adapting pre-trained models to various downstream tasks. However, most prior art indiscriminately uses a fixed prompt distribution across different tasks, neglecting the importance of each block varying depending on the task. In this paper, we introduce adaptive distribution optimization (ADO) by tackling two key questions: (1) How to appropriately and formally define ADO, and (2) How to design an adaptive distribution strategy guided by this definition? Through empirical analysis, we first confirm that properly adjusting the distribution significantly improves VPT performance, and further uncover a key insight that a nested relationship exists between ADO and VPT. Based on these findings, we propose a new VPT framework, termed PRO-VPT (iterative Prompt RelOcation-based VPT), which adaptively adjusts the distribution built upon a nested optimization formulation. Specifically, we develop a prompt relocation strategy derived from this formulation, comprising two steps: pruning idle prompts from prompt-saturated blocks, followed by allocating these prompts to the most prompt-needed blocks. By iteratively performing prompt relocation and VPT, our proposal can adaptively learn the optimal prompt distribution in a nested optimization-based manner, thereby unlocking the full potential of VPT. Extensive experiments demonstrate that our proposal significantly outperforms advanced VPT methods, e.g., PRO-VPT surpasses VPT by 1.6 pp and 2.0 pp average accuracy, leading prompt-based methods to state-of-the-art performance on VTAB-1k and FGVC benchmarks. The code is available at https://github.com/ckshang/PRO-VPT.
BibTeX
@misc{shang2025iterative,
  author = {Chikai Shang and Mengke Li and Yiqun Zhang and Zhen Chen and Jinlin Wu and Fangqing Gu and Yang Lu and Yiu-ming Cheung},
  title = {{Iterative Prompt Relocation for Distribution-Adaptive Visual Prompt Tuning}},
  year = {2025},
  eprint = {2503.06901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06901},
}
Figure