Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

Xu Luo, Ji Zhang, Lianli Gao, Heng Tao Shen, Jingkuan SongarXiv 2026 · arXiv

TL;DR

This paper exposes critical flaws in how we evaluate few-shot transfer learning, namely the "sampling lottery" and the "validation set illusion." To fix this, the authors introduce FEWTRANS, a rigorous 10-dataset benchmark, alongside a label-free Hyperparameter Ensemble (HPE) protocol. Their findings deliver a sobering reality check: the choice of pre-trained model matters far more than the adaptation algorithm, and simple full fine-tuning is often just as good as complex parameter-efficient methods.

What Problem Does This Paper Address?

Few-shot transfer learning—adapting a massive pre-trained model to a new task using only a handful of labeled examples—is highly attractive for real-world applications where data is scarce. However, the sub-field suffers from an evaluation crisis. The authors identify two fatal flaws in current evaluation protocols:

  1. The "Sampling Lottery" Effect: In few-shot scenarios, performance varies wildly depending on which specific samples are randomly picked for the training task. Evaluating a model on just a few tasks can completely flip the ranking of different algorithms.
  2. The "Validation Set Illusion": Standard practices often tune hyperparameters (like learning rates) on a large, target-domain validation set. In a true few-shot scenario, such a validation set simply does not exist. Relying on it creates an unrealistic and unfair benchmark. Traditional cross-validation also breaks down because there isn't enough data to split reliably.
Fig. 1. Average accuracy and 95% confidence intervals of single-task few- shot transfer evaluation of a pretrained DINOv2-s model on EuroSAT [30].

Methods

To address these issues, the authors propose the Hyperparameter Ensemble (HPE) protocol. Instead of trying to hunt down a single "best" hyperparameter configuration using non-existent validation data, HPE trains models across a predefined grid of hyperparameter candidates. At test time, it averages the logits (outputs) produced by all individual configurations:

$$S(x) = \frac{1}{N} \sum_{i=1}^{N} g_{\phi_i}(x)$$

This approach acts as a buffer against poor hyperparameter choices and naturally penalizes volatile, highly sensitive algorithms without requiring any extra labeled validation data.

Additionally, to understand why full fine-tuning performs so well without overfitting, the authors conduct a mechanistic analysis using Centered Kernel Alignment (CKA) to measure feature distribution shifts, alongside $L_2$ norm measurements of parameter updates. They also utilize adjusted Zipf frequency scores to quantify text domain shifts for multimodal models like CLIP.

Datasets

The authors introduce FEWTRANS, a comprehensive benchmark consisting of 10 diverse downstream datasets selected to ensure semantic diversity, data integrity, and genuine algorithmic challenge:

  • ImageNet-Sketch
  • Describable Textures Dataset (DTD)
  • CIFAR-100
  • VGG Flowers
  • UCF-101 (action recognition frames)
  • EuroSAT (satellite imagery)
  • Quick Draw
  • Fungi
  • Plant Disease
  • Aircraft

Unlike older benchmarks, FEWTRANS includes natural language class names (enabling multimodal evaluation), features class-imbalanced task sampling, and implements a strict base-to-novel class split to test generalization.

Key Insights and Ideas

  • Model Scale > Algorithm Choice: The pre-training dataset size and model architecture are the dominant factors driving downstream performance. The gaps between sophisticated transfer algorithms (like LoRA, Adapters, or Prompt Tuning) are often negligible when evaluated rigorously.
  • The Micro-Adjustment Principle: Counter-intuitively, full fine-tuning (Full-FT) doesn't overfit in extreme few-shot settings. The authors show it succeeds through distributed "micro-adjustments" across the entire parameter space (with tiny $L_2$ update magnitudes between 0.01 and 0.07). This keeps the model within the broad, flat minima basin of the pre-trained weights, providing implicit regularization.
  • Linguistic Rarity Causes Multimodal Collapse: Multimodal models like CLIP experience severe performance drops on specialized domains (like Fungi or Plant Disease). Using Zipf frequency scores, the authors show this is a direct result of "textual domain shift"—the text encoder struggles with rare scientific Latin terms or complex compositional phrases, making full visual fine-tuning an indispensable correction mechanism.
Impact of text domain shift on CLIP adaptation, showing a strong negative correlation between text semantic rarity and adaptation gain.

Key Results

  • HPE is Highly Stable: The HPE protocol consistently approximates "Oracle-best" hyperparameter selection and remains remarkably stable regardless of the grid spacing chosen.
  • LoRA vs. Full Fine-Tuning is a Wash: A paired statistical analysis across 6,000 distinct tasks revealed that the practical advantage of LoRA over simple full fine-tuning is minimal. The overall Cohen’s $d$ effect size was only $-0.22$, which is statistically classified as "negligible." On 6 out of 10 datasets, there was no statistically significant difference at all.
  • CLIP Text Bottleneck Quantified: The authors found a powerful negative Spearman rank correlation ($\rho = -0.881$) between text semantic rarity and few-shot adaptation gain, proving that linguistic domain shift is the primary bottleneck for vision-language models.

Limitations and Future Work

The authors note that FEWTRANS currently focuses purely on standard task accuracy. They highlight two major avenues for future extension:

  1. Adversarial Robustness: Evaluating how well few-shot adapted models resist malicious perturbations.
  2. Out-of-Distribution (OOD) Generalization: Testing how a model adapted on a tiny support set generalizes to test sets experiencing severe covariate or domain shifts.

Summary updated

Abstract

Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at https://github.com/Frankluox/FewTrans.
BibTeX
@misc{luo2026benchmarking,
  author = {Xu Luo and Ji Zhang and Lianli Gao and Heng Tao Shen and Jingkuan Song},
  title = {{Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols}},
  year = {2026},
  eprint = {2603.00478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.00478},
}
Figure