Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

Tengxue Zhang, Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, Bin Yang

Abstract

Selecting the optimal pre-trained model (PTM) for a specific downstream task is a critical challenge in transfer learning. Traditional methods rely on expensive fine-tuning or monolithic feature analysis. This paper introduces DISCO (DIstribution of Spectral COmponents), a novel assessment framework that analyzes models through the lens of spectral decomposition. By applying Singular Value Decomposition (SVD) to extracted features, we identify stable, transferable components that drive performance. DISCO leverages the distribution of these components to rank models efficiently. Our approach achieves state-of-the-art results across image classification and object detection benchmarks, offering a unified and scalable solution for model selection.

Introduction: The Model Selection Dilemma

The rapid growth of model hubs has provided practitioners with a vast array of pre-trained models. However, identifying which model will perform best on a new dataset—such as specialized medical imagery or niche object detection—remains difficult. Fine-tuning every candidate is computationally prohibitive, while existing zero-shot metrics often fail to account for the internal feature dynamics that shift during training.

The DISCO framework provides an accurate and efficient alternative to time-consuming fine-tuning for ranking pre-trained models.

As illustrated in Figure 1, there is a clear trade-off between accuracy and efficiency. DISCO bridges this gap by providing a metric that matches the predictive power of fine-tuning at a fraction of the temporal cost.

Methodology: The Spectral Perspective

The core innovation of DISCO is the decomposition of features into spectral components. By performing SVD on the feature matrix $F$, we obtain singular values that represent the "energy" or importance of different directions in the feature space.

Key Observation: Energy Concentration

A pivotal discovery in this research is that not all spectral components are equally transferable. Through empirical analysis, the authors found that components associated with larger singular values tend to be more stable during the fine-tuning process.

Distribution of spectral components ($S_{ratio}$) before and after fine-tuning, showing energy concentration in transferable components.

Figure 3 demonstrates this effect: after fine-tuning, the proportion of energy ($S_{ratio}$) significantly increases in the primary spectral groups. This suggests that a model's inherent spectral distribution is a strong indicator of its transfer potential. DISCO quantifies this by combining the energy ratio ($S_{ratio}$) with a task-specific performance score ($S_{task}$), creating a robust ranking criterion.

Efficiency via Hard-Example Selection

To ensure scalability, DISCO incorporates a "hard-example selection" mechanism. Using Linear Discriminant Analysis (LDA), the framework identifies the most challenging samples for a given model. By performing spectral analysis on only these "hard" samples (typically 10-20% of the dataset), DISCO maintains high ranking accuracy while drastically reducing computational requirements.

Experimental Evaluation

The authors conducted extensive experiments across 11 classification datasets (e.g., CIFAR, Oxford Pets) and 5 object detection datasets (e.g., Blood Cell Detection).

  • Superior Ranking Accuracy: DISCO achieved a weighted Kendall’s $\tau_w$ of 0.739 for supervised models, outperforming established metrics like NCTI (0.726) and LogME.
  • Versatility in Object Detection: Unlike many competitors, DISCO excels in regression-heavy tasks, showing a 21% improvement over the previous state-of-the-art (Etran) in model ranking for object detection.
  • Operational Speed: The method is highly efficient; for instance, it can rank a model pool on the CIFAR-10 dataset in just 32 seconds, compared to the hours required for full fine-tuning.

Conclusion

DISCO represents a significant step forward in pre-trained model assessment. By shifting the focus from monolithic feature blocks to the distribution of spectral components, it provides a theoretically grounded and practically efficient tool for the AI community. This spectral lens not only improves ranking accuracy but also offers a unified framework applicable to a wide variety of computer vision tasks.

Summary updated

Abstract

Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.
BibTeX
@inproceedings{zhang2025assessing,
  author = {Tengxue Zhang and Yang Shu and Xinyang Chen and Yifei Long and Chenjuan Guo and Bin Yang},
  title = {{Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34414},
}
Figure