Abstract
Selecting the optimal pre-trained model (PTM) for a specific downstream task is a critical challenge in transfer learning. Traditional methods rely on expensive fine-tuning or monolithic feature analysis. This paper introduces DISCO (DIstribution of Spectral COmponents), a novel assessment framework that analyzes models through the lens of spectral decomposition. By applying Singular Value Decomposition (SVD) to extracted features, we identify stable, transferable components that drive performance. DISCO leverages the distribution of these components to rank models efficiently. Our approach achieves state-of-the-art results across image classification and object detection benchmarks, offering a unified and scalable solution for model selection.
Introduction: The Model Selection Dilemma
The rapid growth of model hubs has provided practitioners with a vast array of pre-trained models. However, identifying which model will perform best on a new dataset—such as specialized medical imagery or niche object detection—remains difficult. Fine-tuning every candidate is computationally prohibitive, while existing zero-shot metrics often fail to account for the internal feature dynamics that shift during training.
As illustrated in Figure 1, there is a clear trade-off between accuracy and efficiency. DISCO bridges this gap by providing a metric that matches the predictive power of fine-tuning at a fraction of the temporal cost.
Methodology: The Spectral Perspective
The core innovation of DISCO is the decomposition of features into spectral components. By performing SVD on the feature matrix $F$, we obtain singular values that represent the "energy" or importance of different directions in the feature space.
Key Observation: Energy Concentration
A pivotal discovery in this research is that not all spectral components are equally transferable. Through empirical analysis, the authors found that components associated with larger singular values tend to be more stable during the fine-tuning process.
Figure 3 demonstrates this effect: after fine-tuning, the proportion of energy ($S_{ratio}$) significantly increases in the primary spectral groups. This suggests that a model's inherent spectral distribution is a strong indicator of its transfer potential. DISCO quantifies this by combining the energy ratio ($S_{ratio}$) with a task-specific performance score ($S_{task}$), creating a robust ranking criterion.
Efficiency via Hard-Example Selection
To ensure scalability, DISCO incorporates a "hard-example selection" mechanism. Using Linear Discriminant Analysis (LDA), the framework identifies the most challenging samples for a given model. By performing spectral analysis on only these "hard" samples (typically 10-20% of the dataset), DISCO maintains high ranking accuracy while drastically reducing computational requirements.
Experimental Evaluation
The authors conducted extensive experiments across 11 classification datasets (e.g., CIFAR, Oxford Pets) and 5 object detection datasets (e.g., Blood Cell Detection).
- Superior Ranking Accuracy: DISCO achieved a weighted Kendall’s $\tau_w$ of 0.739 for supervised models, outperforming established metrics like NCTI (0.726) and LogME.
- Versatility in Object Detection: Unlike many competitors, DISCO excels in regression-heavy tasks, showing a 21% improvement over the previous state-of-the-art (Etran) in model ranking for object detection.
- Operational Speed: The method is highly efficient; for instance, it can rank a model pool on the CIFAR-10 dataset in just 32 seconds, compared to the hours required for full fine-tuning.
Conclusion
DISCO represents a significant step forward in pre-trained model assessment. By shifting the focus from monolithic feature blocks to the distribution of spectral components, it provides a theoretically grounded and practically efficient tool for the AI community. This spectral lens not only improves ranking accuracy but also offers a unified framework applicable to a wide variety of computer vision tasks.
Summary updated