TL;DR
This paper provides a systematic evaluation of Visual Prompting (VP) against standard transfer learning methods like LoRA and Full Fine-tuning across diverse model architectures, including the new Mamba family. The authors find that while VP is highly memory-efficient, ConvNext architectures currently lead in VP performance, and LoRA remains the superior choice for data-scarce tasks where VP tends to overfit.
What Problem Does This Paper Address?
As vision foundation models grow in size and complexity, the "de facto" standard of Full Fine-tuning (FFT) is becoming prohibitively expensive in terms of compute and memory. This has led to the rise of Parameter-Efficient Fine-Tuning (PEFT) methods.
One emerging technique is Visual Prompting (VP), which adapts a frozen model to a new task by adding learnable perturbations to the input image rather than modifying the model's weights. However, the field lacks a systematic comparison of how VP interacts with different architectural families—specifically traditional Convolutions, Transformers, and the newer Mamba-based state-space models. The authors aim to provide practical guidelines on which architectures and tuning methods work best under resource constraints.
Methods
The authors compare four primary transfer paradigms:
- Visual Prompting (VP): Specifically using the AutoVP framework, which includes learnable prompt resizing and Fully Connected Label Mapping (FCLM) to map source classes to target classes.
- Linear Probing (LP): Training only a new linear classifier on top of frozen features.
- Low Rank Adaptation (LoRA): Injecting low-rank trainable matrices into the model layers.
- Full Fine-tuning (FFT): Updating all model parameters.
These methods were tested across three major architecture families:
- Attention-based: ViT, DeiT, Swin Transformer.
- Convolution-based: ConvNext.
- Mamba-based: VMamba, MambaOut, MambaVision.
Datasets
The models were pretrained on ImageNet-1k and evaluated on a diverse benchmark of eight downstream image classification tasks:
- General: CIFAR-10, CIFAR-100.
- Specialized: Flowers102, OxfordPets, Food101.
- Challenging/Data-Scarce: DTD (Describable Textures Dataset), EuroSAT (Satellite imagery), GTSRB (Traffic signs).
The DTD dataset was highlighted as a particularly difficult task for evaluating overfitting in efficient tuning methods.
Key Insights and Ideas
- ConvNext is the VP King: Despite the hype around Transformers and Mamba, ConvNext-B achieved the highest average accuracy across the benchmark when using VP.
-
Label Mapping > Prompt Resizing: The authors discovered that the significant performance gains in modern VP (like AutoVP) come primarily from the Fully Connected Label Mapping (FCLM) rather than the ability to resize the visual prompt. Ablation study showing FCLM provides the bulk of the performance gain compared to prompt resizing.
-
The Overfitting Trap: VP is prone to overfitting on small datasets. On the DTD dataset, models reached >99% training accuracy but lagged significantly behind LoRA and LP in test accuracy.
- Pretraining Accuracy as a Predictor: Generally, a model's original ImageNet-1k accuracy correlates well with its downstream VP performance, though this relationship weakens or even reverses on specific tasks like GTSRB or EuroSAT. Correlation between ImageNet pretraining accuracy and average VP performance.
Key Results
- Memory Efficiency: VP and LP are remarkably lightweight, requiring only ~15% of the GPU memory needed for Full Fine-tuning. For a VMamba-B model, VP used ~12.5 GB of VRAM compared to ~72.3 GB for FFT.
-
LoRA's Dominance: On the challenging DTD dataset, LoRA outperformed FFT for five out of six models while training significantly fewer parameters. Comparison of different transfer learning paradigms on the DTD dataset.
-
Regularization Helps VP: Applying L2 regularization specifically to the label mapping weights was found to be the most effective way to boost VP's generalization, though it still couldn't close the gap to LoRA on data-scarce tasks.
Limitations and Future Work
- Mamba Optimization: The authors note that their LoRA implementation for Mamba-based models might be less optimized than the well-established implementations for Transformers, potentially skewing results slightly.
- Task Variety: The study focused exclusively on image classification. Future work could explore how these findings translate to more complex tasks like object detection or segmentation.
- VP vs. LP: In extremely resource-constrained settings, Linear Probing (LP) often remains superior to VP, suggesting that VP still needs architectural or algorithmic refinements to become the go-to "frozen" adaptation method.
Summary updated