Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?

Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, Dongfang Liu

TL;DR

This paper tackles the "elephant in the room" for computer vision researchers: when and why should you use Visual Prompt Tuning (VPT) instead of traditional full finetuning? Through an extensive study of 19 datasets, the authors reveal that VPT isn't just a "budget" option—it actually outperforms full finetuning in scenarios where task objectives shift drastically or data distributions remain highly similar, primarily by preserving valuable pretrained features that finetuning tends to destroy.

Introduction

As vision models like Vision Transformers (ViT) scale to billions of parameters, "Full Finetuning" (updating every single weight in the model) has become a double-edged sword. While powerful, it is computationally expensive, storage-intensive, and often leads to "catastrophic forgetting," where the model loses the general knowledge it gained during pretraining.

Visual Prompt Tuning (VPT) emerged as a parameter-efficient alternative. Instead of touching the model's weights, you keep the backbone frozen and only train a small set of "prompts"—learnable vectors added to the input space. While VPT has shown impressive results, the community lacked a clear understanding of the specific conditions under which it beats finetuning and the underlying mechanisms that make it work. This paper aims to provide that roadmap.

Methodology and Experimental Setup

The authors conduct a massive "head-to-head" comparison between VPT and Full Finetuning (FT) across 19 diverse datasets. To get to the "why" and "when," they categorize transfer learning scenarios into four quadrants based on two dimensions:

  1. Task Objective Disparity: How different is the new task (e.g., counting objects) from the original pretraining task (e.g., classifying objects)?
  2. Data Distribution Similarity: How similar is the new image data (e.g., satellite photos vs. natural photos) to the pretraining data?
Transfer learning scenarios categorized by data and task similarity

The study primarily utilizes the VTAB-1k (Visual Transfer Learning Benchmark), which consists of 19 tasks grouped into three categories: Natural (standard images), Specialized (medical/satellite imagery), and Structured (tasks requiring geometric understanding). All models were pretrained on ImageNet-21k.

To test optimization hypotheses, they also introduce two hybrid training strategies: "Mixed" (adding prompts and finetuning the whole network) and "FT-then-PT" (finetuning the network first, then adding prompts). Explainability tools like GradCAM and Integrated Gradients are used to visualize how the model's "focus" changes.

Key Findings

The Three-Quadrant Rule

VPT is generally preferable in 3 out of 4 transfer learning scenarios. It excels when there is a high disparity in task objectives (e.g., transitioning from classification to counting) or when the data distribution is very similar to the pretraining set (e.g., both involve natural images). Overall, VPT outperformed Full Finetuning in 16 out of 19 instances on the VTAB-1k benchmark.

Structured Tasks and Feature Preservation

The biggest win for VPT was on "Structured" tasks requiring geometric comprehension. Here, VPT achieved a mean accuracy of 56.09%, compared to just 48.07% for Full Finetuning.

The "secret sauce" of VPT isn't just that it has fewer parameters to overfit. Rather, it's that VPT preserves the original feature space. Full finetuning often "distorts" the backbone's well-learned features to fit a small downstream dataset, which can actually hurt performance. Visual evidence from GradCAM showed that VPT tends to produce much cleaner, more "concept-focused" heatmaps compared to FT.

Data Scaling: The Crossover Point

VPT is significantly more robust when data is scarce. In one-shot learning (training on just one example per class), VPT outperformed FT in 13 out of 19 tasks. However, the authors found a "crossover point." As the amount of downstream training data increases, Full Finetuning eventually closes the gap and can surpass VPT.

Accuracy comparison across varying training set sizes

Limitations and Future Work

  • The Data Ceiling: The study confirms that if you have a massive amount of task-specific data, Full Finetuning is still the gold standard. VPT's advantages are most pronounced in the "small-to-medium" data regime.
  • Beyond Recognition: This paper focuses heavily on image classification. The dynamics of VPT in more complex tasks like semantic segmentation or image captioning require further dedicated study.
  • Theoretical Gap: While the paper provides strong empirical evidence for feature preservation, a rigorous theoretical explanation of the "non-obvious pathway" through which prompts interact with frozen features remains an open challenge.

Summary updated

Abstract

As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning. However, the conditions favoring VPT (the "when") and the underlying rationale (the "why") remain unclear. In this paper, we conduct a comprehensive analysis across 19 distinct datasets and tasks. To understand the "when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives ($e.g.$, transitioning from classification to counting), or 2) a notable similarity in data distributions between the two tasks ($e.g.$, both involve natural images). In exploring the "why" dimension, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.
BibTeX
@inproceedings{han2024facing,
  author = {Cheng Han and Qifan Wang and Yiming Cui and Wenguan Wang and Lifu Huang and Siyuan Qi and Dongfang Liu},
  title = {{Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=bJx4iOIOxn},
}
Figure