SMS: An Efficient Source Model Selection Framework for Model Reuse

Minjun Zhao, Lu Chen, Keyu Yang, Yuntao Du, Yunjun Gao

Abstract

Materialized model query aims to find the most appropriate materialized model as the initial model for model reuse. It is the precondition of model reuse, and has recently attracted much attention. Nonetheless, the existing methods suffer from the need to provide source data, limited range of applications, and inefficiency since they do not construct a suitable metric to measure the target-related knowledge of materialized models. To address this, we present MMQ, a source-data free, general, efficient, and effective materialized model query framework. It uses a Gaussian mixture-based metric called separation degree to rank materialized models. For each materialized model, MMQ first vectorizes the samples in the target dataset into probability vectors by directly applying this model, then utilizes Gaussian distribution to fit for each class of probability vectors, and finally uses separation degree on the Gaussian distributions to measure the target-related knowledge of the materialized model. Moreover, we propose an improved MMQ (I-MMQ), which significantly reduces the query time while retaining the query performance of MMQ. Extensive experiments on a range of practical model reuse workloads demonstrate the effectiveness and efficiency of MMQ.

1. Introduction

In modern machine learning, developers rarely train models from scratch. Instead, they download a "materialized" (pre-trained) model from a repository like GitHub or HuggingFace and fine-tune it on their specific dataset. However, as repositories grow to house thousands of models, a new problem emerges: how do you know which model will fine-tune best for your specific task?

Brute-force retraining every candidate model is computationally impossible. Existing selection methods often require the source data the model was originally trained on—which is frequently private or too massive to share—or they fail if the source and target tasks have completely different labels. To overcome these barriers, we introduce the MMQ framework.

2. The MMQ Framework

MMQ is a source-data-free, zero-shot framework designed to rank models based on their "target-related knowledge." The core hypothesis is that a model that will fine-tune well already acts as a good feature extractor for the new data, naturally separating different classes in its latent space.

The MMQ Framework workflow, illustrating the transition from target dataset vectorization to the computation of the separation degree.

As shown in the framework overview, MMQ operates in three primary steps:

  1. Vectorization (Zero-Shot Inference): The target dataset $D_t$ is passed through a candidate model $M_i$ without any training. The model's final layer outputs are normalized using an "extended softmax" function to create "soft labels."
  2. Fitting: These soft labels are grouped by their ground-truth classes in the target dataset. MMQ then fits a multivariate Gaussian distribution to each cluster.
  3. Computation (Separation Degree): MMQ calculates the "separation degree" between these Gaussian distributions. If the Gaussians are highly separated, it indicates the model naturally distinguishes the new classes well.

3. Measuring Model Quality via Separation Degree

The "Separation Degree" (SD) is the mathematical heart of the framework. It measures how distinct the feature distributions of different classes are when processed by a specific model.

Visual representation of the Separation Degree (SD) metric between Gaussian distributions, ranging from complete separation (SD=1.0) to complete overlap (SD=0.0).

As illustrated above, a high SD (approaching 1.0) suggests that the model's pre-trained features are highly relevant to the target task, even if the labels differ. Conversely, an SD of 0.0 indicates that the model cannot distinguish between the target classes, making it a poor candidate for reuse.

4. Efficiency Improvements: I-MMQ

Calculating the separation degree involves matrix inversions that scale cubically $O(P^3)$ with the number of output dimensions $P$. To address this bottleneck, we introduce I-MMQ (Improved MMQ). I-MMQ adds a random, untrained fully connected layer to the end of the model to project the outputs down to a much smaller dimension (e.g., 25). This drastically speeds up the computation while preserving the relative structural separation of the clusters.

5. Experimental Evaluation

We evaluated MMQ across image classification, text classification, and image regression tasks using datasets such as ImageNet-2012, CUB-200-2011, and IMDB. Key results include:

  • High Correlation: MMQ and I-MMQ consistently achieved a higher Pearson Correlation Coefficient (PCC) with actual fine-tuning accuracy compared to state-of-the-art baselines. On ImageNet subsets, MMQ achieved a PCC of 0.791, significantly outperforming the CAS baseline (0.549).
  • Computational Speed: I-MMQ drastically reduces overhead. In ImageNet experiments, mathematical computation time dropped from 2.6 hours to just 0.6 hours while maintaining ranking accuracy.
  • Architecture Agnostic: The framework successfully ranked a mixed set of 72 models spanning 12 different architectures (e.g., ResNet, VGG, MobileNet), proving its generalizability.

6. Conclusion

MMQ provides an efficient, source-data-free solution to the materialized model query problem. By leveraging the separation degree of zero-shot feature distributions, it identifies the most promising models for transfer learning without the need for expensive retraining or access to private source data. Future work will explore the framework's performance on emerging paradigms like prompt-tuning and non-Gaussian data distributions.

Summary updated

Abstract

Materialized model query aims to find the most appropriate materialized model as the initial model for model reuse. It is the precondition of model reuse, and has recently attracted much attention. {Nonetheless, the existing methods suffer from the need to provide source data, limited range of applications, and inefficiency since they do not construct a suitable metric to measure the target-related knowledge of materialized models. To address this, we present \textsf{MMQ}, a source-data free, general, efficient, and effective materialized model query framework.} It uses a Gaussian mixture-based metric called separation degree to rank materialized models. For each materialized model, \textsf{MMQ} first vectorizes the samples in the target dataset into probability vectors by directly applying this model, then utilizes Gaussian distribution to fit for each class of probability vectors, and finally uses separation degree on the Gaussian distributions to measure the target-related knowledge of the materialized model. Moreover, we propose an improved \textsf{MMQ} (\textsf{I-MMQ}), which significantly reduces the query time while retaining the query performance of \textsf{MMQ}. Extensive experiments on a range of practical model reuse workloads demonstrate the effectiveness and efficiency of \textsf{MMQ}.
BibTeX
@article{zhao2021sms,
  author = {Minjun Zhao and Lu Chen and Keyu Yang and Yuntao Du and Yunjun Gao},
  title = {{SMS: An Efficient Source Model Selection Framework for Model Reuse}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2021},
}
Figure