Abstract
Traditional transfer learning often assumes that the best-performing model on a source task will yield the best results on a target task. However, empirical evidence suggests that transferability peaks at different stages of pre-training and can even degrade as the model over-specializes to the source domain. We introduce Fréchet Pre-train Distance (FPD), a metric that measures the statistical distance between source and target datasets in the feature space of a pre-trained network. FPD serves as a reliable, low-cost proxy for transfer performance, enabling the selection of optimal checkpoints without exhaustive fine-tuning. Our analysis reveals critical phenomena such as the "Transfer Peak" and the "AlexNet Anomaly," where intermediate weights outperform fully converged ones.
Introduction
In modern AI, transfer learning is the standard: a model pre-trained on a massive dataset (like ImageNet) is fine-tuned for a specific, smaller task. Usually, engineers select the final, highest-performing checkpoint from the source task. This paper identifies a major gap in this practice: the best source model is not always the best transfer model. In some cases, a model still learning the source task transfers better than one that has fully converged. Picking the wrong checkpoint can even lead to results inferior to training from scratch. To solve this without the massive computational cost of fine-tuning every checkpoint, we propose a predictive metric to identify optimal transferability.
Methodology: Fréchet Pre-train Distance
The authors propose the Fréchet Pre-train Distance (FPD), inspired by the Fréchet Inception Distance (FID) used to evaluate GANs. The intuition is that if a pre-trained network perceives the source and target datasets as having similar statistical distributions in its internal feature space, it is likely to transfer well.
- Feature Extraction: For a given checkpoint, source and target data are passed through the network to obtain feature representations.
- Statistical Modeling: These features are modeled as multivariate Gaussian distributions, calculating their respective means and covariances.
- Distance Calculation: The Fréchet distance between these two distributions is computed.
A lower FPD indicates higher similarity and, consequently, higher predicted transferability. The authors also explore Spectral Normalization (SN) to stabilize weights and prevent the over-parameterization that often ruins transferability in early training stages.
Experimental Setup
The study utilized standard computer vision benchmarks:
- Source Datasets: CIFAR-100 and SVHN (Street View House Numbers).
- Target Datasets: CIFAR-10 and MNIST.
- Architectures: AlexNet, VGG-16, and various ResNet flavors (18, 50, 101).
To simulate realistic "small-data" scenarios, the researchers used reduced versions of target sets (e.g., 10% of CIFAR-10 or 1% of MNIST).
Transfer Dynamics and the AlexNet Anomaly
Experimental results reveal complex transfer dynamics across different architectures. A notable finding is the "AlexNet Anomaly": in early training stages, AlexNet's transfer performance often dips significantly below the baseline of training from scratch. This is attributed to its massive, over-parameterized fully connected layers which initially learn noisy or overly specific features before stabilizing.
Key Results
- The Transfer Peak: Transfer performance often peaks before the source task finishes training. As the model continues to train, it may overfit the source domain, causing transfer performance to drop. For example, in ResNet-18, a clear peak is observed followed by a decline in transfer accuracy.
- Predictive Power of FPD: FPD successfully identified optimal checkpoints across different architectures and learning rates. On the CIFAR100-to-CIFAR10 task using AlexNet, transfer accuracy dropped by as much as 5 percentage points if the final checkpoint was used instead of the optimal early checkpoint identified by FPD.
Conclusion
FPD provides a quantifiable method to select pre-trained checkpoints, saving significant computational resources. While currently limited by the Gaussian assumption and a focus on image classification, it offers a robust framework for understanding and optimizing transfer learning. Future work will investigate its applicability to complex tasks like object detection and non-visual domains like Natural Language Processing.
Summary updated