A note on data-efficient finetuning of vision foundation models

Niclas Hergenröther, Antonio OrvietoarXiv 2026 · arXiv

TL;DR

This paper provides a systematic evaluation of Visual Prompting (VP) against standard transfer learning methods like LoRA and Full Fine-tuning across diverse model architectures, including the new Mamba family. The authors find that while VP is highly memory-efficient, ConvNext architectures currently lead in VP performance, and LoRA remains the superior choice for data-scarce tasks where VP tends to overfit.

What Problem Does This Paper Address?

As vision foundation models grow in size and complexity, the "de facto" standard of Full Fine-tuning (FFT) is becoming prohibitively expensive in terms of compute and memory. This has led to the rise of Parameter-Efficient Fine-Tuning (PEFT) methods.

One emerging technique is Visual Prompting (VP), which adapts a frozen model to a new task by adding learnable perturbations to the input image rather than modifying the model's weights. However, the field lacks a systematic comparison of how VP interacts with different architectural families—specifically traditional Convolutions, Transformers, and the newer Mamba-based state-space models. The authors aim to provide practical guidelines on which architectures and tuning methods work best under resource constraints.

Methods

The authors compare four primary transfer paradigms:

  1. Visual Prompting (VP): Specifically using the AutoVP framework, which includes learnable prompt resizing and Fully Connected Label Mapping (FCLM) to map source classes to target classes.
  2. Linear Probing (LP): Training only a new linear classifier on top of frozen features.
  3. Low Rank Adaptation (LoRA): Injecting low-rank trainable matrices into the model layers.
  4. Full Fine-tuning (FFT): Updating all model parameters.

These methods were tested across three major architecture families:

  • Attention-based: ViT, DeiT, Swin Transformer.
  • Convolution-based: ConvNext.
  • Mamba-based: VMamba, MambaOut, MambaVision.

Datasets

The models were pretrained on ImageNet-1k and evaluated on a diverse benchmark of eight downstream image classification tasks:

  • General: CIFAR-10, CIFAR-100.
  • Specialized: Flowers102, OxfordPets, Food101.
  • Challenging/Data-Scarce: DTD (Describable Textures Dataset), EuroSAT (Satellite imagery), GTSRB (Traffic signs).

The DTD dataset was highlighted as a particularly difficult task for evaluating overfitting in efficient tuning methods.

Key Insights and Ideas

  • ConvNext is the VP King: Despite the hype around Transformers and Mamba, ConvNext-B achieved the highest average accuracy across the benchmark when using VP.
  • Label Mapping > Prompt Resizing: The authors discovered that the significant performance gains in modern VP (like AutoVP) come primarily from the Fully Connected Label Mapping (FCLM) rather than the ability to resize the visual prompt. Ablation study showing FCLM provides the bulk of the performance gain compared to prompt resizing.

  • The Overfitting Trap: VP is prone to overfitting on small datasets. On the DTD dataset, models reached >99% training accuracy but lagged significantly behind LoRA and LP in test accuracy.

  • Pretraining Accuracy as a Predictor: Generally, a model's original ImageNet-1k accuracy correlates well with its downstream VP performance, though this relationship weakens or even reverses on specific tasks like GTSRB or EuroSAT. Correlation between ImageNet pretraining accuracy and average VP performance.
Figure 3: Flowers102 performance of VMamba-T for com- binations of ILM and FCLM with and without enabled re- sizing.
Figure 1: ImageNet1k-pretrain and average VP top-1 accu- racy over the tasks described in Section 3. We follow the recipe of Tsao et al. (2024). All models are tuned to best. Dotted line is linear regression, full results in Table 1.

Key Results

  • Memory Efficiency: VP and LP are remarkably lightweight, requiring only ~15% of the GPU memory needed for Full Fine-tuning. For a VMamba-B model, VP used ~12.5 GB of VRAM compared to ~72.3 GB for FFT.
  • LoRA's Dominance: On the challenging DTD dataset, LoRA outperformed FFT for five out of six models while training significantly fewer parameters. Comparison of different transfer learning paradigms on the DTD dataset.

  • Regularization Helps VP: Applying L2 regularization specifically to the label mapping weights was found to be the most effective way to boost VP's generalization, though it still couldn't close the gap to LoRA on data-scarce tasks.

Comparison of different transfer learning paradigms on the DTD dataset.

Limitations and Future Work

  • Mamba Optimization: The authors note that their LoRA implementation for Mamba-based models might be less optimized than the well-established implementations for Transformers, potentially skewing results slightly.
  • Task Variety: The study focused exclusively on image classification. Future work could explore how these findings translate to more complex tasks like object detection or segmentation.
  • VP vs. LP: In extremely resource-constrained settings, Linear Probing (LP) often remains superior to VP, suggesting that VP still needs architectural or algorithmic refinements to become the go-to "frozen" adaptation method.

Summary updated

Abstract

Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.
BibTeX
@misc{hergenrother2026note,
  author = {Niclas Hergenröther and Antonio Orvieto},
  title = {{A note on data-efficient finetuning of vision foundation models}},
  year = {2026},
  eprint = {2610.37510},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37510},
}
Figure