Abstract: With the explosive increase of big data, training a Machine Learning (ML) model becomes a computation-intensive workload, which would take days or even weeks. Thus, reusing an already trained model has received attention, which is called transfer learning. Transfer learning avoids training a new model from scratch by transferring knowledge from a source task to a target task. Existing transfer learning methods mostly focus on how to improve the performance of the target task through a specific source model, and assume that the source model is given. Although many source models are available, it is difficult for data scientists to select the best source model for the target task manually. Hence, how to efficiently select a suitable source model in a model database for model reuse is an interesting but unsolved problem. In this paper, we propose SMS (later referred to as the MMQ framework), an effective, efficient, and flexible source model selection framework. SMS is effective even when the source and target datasets have significantly different data labels, and is flexible to support source models with any type of structure, and is efficient to avoid any training process. For each source model, SMS first vectorizes the samples in the target dataset into soft labels by directly applying this model to the target dataset, then uses Gaussian distributions to fit for clusters of soft labels, and finally measures the distinguishing ability of the source model using Gaussian mixture-based metric. Moreover, we present an improved SMS (I-SMS), which decreases the output number of the source model. I-SMS can significantly reduce the selection time while retaining the selection performance of SMS. Extensive experiments on a range of practical model reuse workloads demonstrate the effectiveness and efficiency of SMS.
Introduction
In the era of "Foundation Models" and massive model zoos like TensorFlow Hub or PyTorch Hub, the first step of transfer learning—selecting the right source model—is often overlooked. Most researchers assume the starting model is given, but in practice, choosing the wrong one can lead to "negative transfer," where the pre-trained knowledge actually hinders the new task.
Existing selection methods suffer from three major flaws: dependency on source data (which is often private or too large), lack of generality (requiring matching labels between source and target), and extreme computational inefficiency (requiring partial retraining of thousands of candidates). The Materialized Model Query (MMQ) framework addresses these by allowing developers to find the best pre-trained model for a new task without needing the original training data or performing expensive retraining.
The MMQ Framework
MMQ is a source-data-free framework that treats candidate models as feature extractors. The process follows three main steps to identify which candidate models possess the most relevant knowledge for a target dataset:
- Vectorization: The target dataset $D_t$ is passed through a candidate model $M_i$ (without any training). The model outputs "soft labels" (probability vectors) for each sample. An "extended softmax" function is used to ensure these vectors capture enough nuance for the metric.
- Fitting: Samples are grouped by their actual target labels. For each class, MMQ fits a multivariate Gaussian distribution to the resulting soft labels.
- Computation (Separation Degree): The core innovation is the Separation Degree (SD) metric. If a model is well-suited for the target task, the Gaussian distributions for different classes will be far apart. If the model is "clueless," the distributions will overlap heavily. MMQ ranks models based on this SD score.
To handle models with massive output layers, the authors introduced I-MMQ (Improved MMQ). It uses a random, non-trained fully connected layer to project high-dimensional outputs into a smaller space (e.g., 25 dimensions), drastically reducing the computational cost of matrix operations from $O(P^3)$ to $O(N + m^2)$.
Separation Degree: The Core Metric
The "aha moment" of this research is that a model doesn't need to know the target labels to be useful. If its internal features can naturally separate the target classes into distinct clusters, it already possesses the "discriminative power" needed for successful transfer learning.
As shown in the figure above, when the Gaussian distributions $G_1(x)$ and $G_2(x)$ for two different classes are perfectly separated, the SD is 1.0. As they overlap, the SD decreases. Because MMQ looks at the distribution of outputs rather than specific class predictions, it works even when transferring from a model trained on "dogs and cats" to a task involving "trucks and planes."
Experimental Evaluation
The framework was evaluated across diverse tasks using nine public datasets, including ImageNet-2012, CUB-200-2011, and text classification sets like IMDB. Key results include:
- Accuracy: MMQ consistently achieved higher Pearson Correlation Coefficients (PCC) compared to state-of-the-art methods like CAS and REAPER, meaning its rankings closely matched the actual performance of models after full retraining.
- Speed: MMQ is significantly faster than brute-force approaches. I-MMQ further reduced computation time by nearly 75% without sacrificing ranking quality.
- Versatility: Unlike previous methods limited to CNNs, MMQ showed strong performance on RNNs for text and even handled regression tasks by discretizing continuous labels.
Conclusion
MMQ provides an effective, efficient, and flexible solution for the model selection problem in large-scale model databases. By leveraging the separation degree of soft labels, it bypasses the need for source data and expensive retraining. Future work may explore its application in purely unsupervised scenarios and its compatibility with newer architectures like Vision Transformers (ViTs).
Summary updated