Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

Tengxue Zhang, Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, Bin Yang

Introduction

The proliferation of "Model Hubs" has revolutionized deep learning, offering researchers thousands of pre-trained models. However, selecting the optimal model for a specific target task remains a significant challenge. While fine-tuning every candidate model is the most accurate method for ranking, it is computationally prohibitive. Existing proxy metrics, such as LogME or LEEP, often fail to capture the internal feature dynamics that drive transferability, treating features as monolithic blocks.

The DISCO Framework

To address these limitations, we present DISCO (DIstribution of Spectral COmponents), a novel metric designed to efficiently predict model performance. DISCO moves beyond simple feature separation by analyzing the internal spectral structures of extracted features.

The DISCO framework provides an accurate and efficient alternative to exhaustive fine-tuning for model ranking.

The methodology involves four key steps:

  1. Spectral Decomposition: Applying Singular Value Decomposition (SVD) to features extracted from the pre-trained model to identify distinct spectral components based on their singular values.
  2. Transferability Discovery: Identifying components with larger singular values, which empirical evidence suggests are more stable and transferable throughout the fine-tuning process.
  3. Metric Calculation: The DISCO score is computed as a weighted sum: $S_{DISCO} = \sum (S_{task} \times S_{ratio})$, where $S_{ratio}$ represents the component's weight (energy) and $S_{task}$ measures its performance on the target task using simple classifiers or regressors.
  4. Hard-Example Selection: To enhance efficiency on massive datasets, the analysis is performed on a subset of "difficult" samples that better reflect the model's true transferability limits.

The Spectral Hypothesis

The core innovation of DISCO lies in the observation that not all features contribute equally to transfer learning. Features associated with larger singular values form the "backbone" of the model's representational power.

Empirical analysis of spectral components ($S_{ratio}$) and their relative change ($C_F$) across different groups during fine-tuning for Flowers and Pets datasets.

As shown in the empirical analysis above, fine-tuning naturally filters the feature space, causing the distribution of singular values ($S_{ratio}$) to become more concentrated in the top components (Group 1). DISCO effectively predicts this outcome by rewarding models that already possess a strong concentration of energy in these stable, high-value components. This "fine-tuning as a filter" perspective allows DISCO to anticipate final performance without the actual temporal cost of training.

Experimental Evaluation

DISCO was validated across 11 image classification datasets and 5 object detection benchmarks, utilizing both supervised and self-supervised model pools.

  • State-of-the-Art Accuracy: On supervised classification, DISCO achieved an average Weighted Kendall’s $\tau$ of 0.739, beating previous benchmarks. On fine-grained datasets like Aircraft and Cars, the improvement exceeded 30%.
  • Object Detection Breakthrough: The framework demonstrated remarkable versatility, showing a 21% improvement over the previous state-of-the-art (ETran) in ranking models for regression-heavy detection tasks.
  • Efficiency: By utilizing hard-example selection at a 10% sampling rate, DISCO maintains high ranking accuracy while remaining competitive with the fastest existing metrics like LogME.

Conclusion and Future Work

DISCO provides a robust, theoretically grounded, and efficient approach to model selection. By focusing on the distribution of spectral components, it captures the nuances of transfer learning that previous metrics overlooked. While currently focused on computer vision, future research will explore the applicability of spectral distribution analysis to other modalities, such as Large Language Models (LLMs), and optimize SVD for extremely high-dimensional feature spaces.

Summary updated

Abstract

Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.
BibTeX
@inproceedings{zhang2025assessing,
  author = {Tengxue Zhang and Yang Shu and Xinyang Chen and Yifei Long and Chenjuan Guo and Bin Yang},
  title = {{Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34414},
}
Figure