Introduction
The proliferation of "Model Hubs" has revolutionized deep learning, offering researchers thousands of pre-trained models. However, selecting the optimal model for a specific target task remains a significant challenge. While fine-tuning every candidate model is the most accurate method for ranking, it is computationally prohibitive. Existing proxy metrics, such as LogME or LEEP, often fail to capture the internal feature dynamics that drive transferability, treating features as monolithic blocks.
The DISCO Framework
To address these limitations, we present DISCO (DIstribution of Spectral COmponents), a novel metric designed to efficiently predict model performance. DISCO moves beyond simple feature separation by analyzing the internal spectral structures of extracted features.
The methodology involves four key steps:
- Spectral Decomposition: Applying Singular Value Decomposition (SVD) to features extracted from the pre-trained model to identify distinct spectral components based on their singular values.
- Transferability Discovery: Identifying components with larger singular values, which empirical evidence suggests are more stable and transferable throughout the fine-tuning process.
- Metric Calculation: The DISCO score is computed as a weighted sum: $S_{DISCO} = \sum (S_{task} \times S_{ratio})$, where $S_{ratio}$ represents the component's weight (energy) and $S_{task}$ measures its performance on the target task using simple classifiers or regressors.
- Hard-Example Selection: To enhance efficiency on massive datasets, the analysis is performed on a subset of "difficult" samples that better reflect the model's true transferability limits.
The Spectral Hypothesis
The core innovation of DISCO lies in the observation that not all features contribute equally to transfer learning. Features associated with larger singular values form the "backbone" of the model's representational power.
As shown in the empirical analysis above, fine-tuning naturally filters the feature space, causing the distribution of singular values ($S_{ratio}$) to become more concentrated in the top components (Group 1). DISCO effectively predicts this outcome by rewarding models that already possess a strong concentration of energy in these stable, high-value components. This "fine-tuning as a filter" perspective allows DISCO to anticipate final performance without the actual temporal cost of training.
Experimental Evaluation
DISCO was validated across 11 image classification datasets and 5 object detection benchmarks, utilizing both supervised and self-supervised model pools.
- State-of-the-Art Accuracy: On supervised classification, DISCO achieved an average Weighted Kendall’s $\tau$ of 0.739, beating previous benchmarks. On fine-grained datasets like Aircraft and Cars, the improvement exceeded 30%.
- Object Detection Breakthrough: The framework demonstrated remarkable versatility, showing a 21% improvement over the previous state-of-the-art (ETran) in ranking models for regression-heavy detection tasks.
- Efficiency: By utilizing hard-example selection at a 10% sampling rate, DISCO maintains high ranking accuracy while remaining competitive with the fastest existing metrics like LogME.
Conclusion and Future Work
DISCO provides a robust, theoretically grounded, and efficient approach to model selection. By focusing on the distribution of spectral components, it captures the nuances of transfer learning that previous metrics overlooked. While currently focused on computer vision, future research will explore the applicability of spectral distribution analysis to other modalities, such as Large Language Models (LLMs), and optimize SVD for extremely high-dimensional feature spaces.
Summary updated