TL;DR
This paper exposes critical flaws in how Source Independent Transferability Estimation (SITE) metrics—tools used to select the best pre-trained model for a new task—are currently evaluated. The authors demonstrate that standard benchmarks use unrealistic model spaces and static hierarchies, allowing a naive, dataset-agnostic size heuristic to easily outperform sophisticated, state-of-the-art metrics.
What Problem Does This Paper Address?
When building deep learning applications, starting with a model pre-trained on a massive dataset (like ImageNet) is standard practice. However, choosing the right pre-trained model from a vast "model zoo" is challenging. Fully fine-tuning dozens of candidate models to see which one performs best on your target task is computationally prohibitive.
To solve this, researchers developed Source Independent Transferability Estimation (SITE) metrics. These metrics analyze a candidate model's extracted features on a target dataset to cheaply and quickly predict its downstream performance without full fine-tuning or access to the original source data.
While many new SITE metrics have been proposed, this paper investigates a major blind spot: the benchmarks used to evaluate these metrics are fundamentally broken. Current evaluation protocols create an illusion of high performance, masking the fact that these metrics fail in realistic deployment scenarios.
Methods
The authors critically evaluate the standard SITE benchmarking paradigm using three main experimental approaches:
- Model Ablation: They progressively remove the largest, overrepresented models (e.g., ResNet-152, DenseNet-201) from the benchmark's candidate pool. This tests whether SITE metrics can actually distinguish subtle architectural advantages among models of similar sizes, or if they just rely on identifying the biggest model.
- Static Ranking Heuristic: They introduce a baseline "ranker" that completely ignores the target dataset and simply ranks models based on a fixed sequence of model size and family alternation.
- Pairwise Difference Correlation (Fidelity): They measure the Pearson correlation between the difference in actual downstream accuracy ($\Delta Acc$) and the difference in predicted SITE scores ($\Delta T$) for pairs of models. This determines if the magnitude of a metric's score is practically meaningful.
Datasets
The authors evaluate popular SITE metrics (including LogME, TransRate, SFDA, H-Score, NLEEP, and GBC) across a widely adopted standard benchmark consisting of:
- Target Datasets: CIFAR10, CIFAR100, Oxford Pets, FGVC Aircraft, Food-101, and the Describable Textures Dataset (DTD).
- Candidate Models: 11 pre-trained models spanning families like ResNet, DenseNet, MobileNet, Inceptionv3, MNASNet, and GoogleNet.
Key Insights and Ideas
- The "Big Model" Shortcut: Standard benchmarks mix massive models with tiny, edge-oriented models (like MobileNet). Because larger models predictably perform better across almost all tasks, SITE metrics look highly accurate simply because they successfully detect that a large model is large, rather than understanding task-specific suitability.
- The Naive Baseline Beats the State-of-the-Art: A completely blind, static ranking heuristic based purely on model size dramatically outperforms every single sophisticated SITE metric on the standard benchmark. This reveals that the benchmark rewards leaderboard memorization rather than actual transferability estimation.
- Brittle Performance: When the oversized models are removed from the pool, the performance of nearly all SITE metrics collapses. They cannot reliably predict transferability when forced to compare models of similar computational budgets.
- Meaningless Score Gaps (Lack of Fidelity): For an engineer, a metric score should signify how much better one model is over another. The authors find that current SITE metrics lack this property entirely; a minor accuracy gap can yield a massive score difference, and vice versa.
Key Results
- The Static Ranking Heuristic achieved an average weighted Kendall’s Tau ($\tau_w$) of 0.91 across all datasets, significantly outperforming the best-performing sophisticated metric, LogME, which averaged only 0.57 (see Table 1 in the paper).
- In the model ablation experiments, removing just a few large models caused most metrics to plummet below a $\tau_w$ of 0.6, demonstrating their reliance on flawed benchmark design.
- The fidelity analysis showed a remarkably weak correlation between metric score gaps and actual accuracy gaps across almost all methods, rendering the raw scores functionally uninterpretable for end-users.
Limitations and Future Work
While this work convincingly dismantles current computer vision benchmarks for SITE, the authors note a few limitations and areas for future expansion:
- Domain Scope: The empirical analysis focused primarily on image classification, though the authors highlight that similar static leaderboard issues are beginning to plague NLP and object detection benchmarks.
- Hyperparameter Neglect: Current SITE evaluation protocols assume a fixed fine-tuning pipeline. They do not account for how different optimizers, learning rates, or augmentation strategies might alter the actual ground-truth rankings of the models.
To move the field forward, the authors propose a SITE Benchmark and Evaluation Checklist, urging future researchers to match computational budgets, ensure architectural diversity (e.g., mixing CNNs, ViTs, and MLPs), and engineer benchmarks specifically for high rank dispersion.
Summary updated