Analyzing Deep Neural Network's Transferability via Fréchet Distance

Yifan Ding, Liqiang Wang, Boqing Gong

Abstract

Traditional transfer learning often assumes that the best-performing model on a source task will yield the best results on a target task. However, empirical evidence suggests that transferability peaks at different stages of pre-training and can even degrade as the model over-specializes to the source domain. We introduce Fréchet Pre-train Distance (FPD), a metric that measures the statistical distance between source and target datasets in the feature space of a pre-trained network. FPD serves as a reliable, low-cost proxy for transfer performance, enabling the selection of optimal checkpoints without exhaustive fine-tuning. Our analysis reveals critical phenomena such as the "Transfer Peak" and the "AlexNet Anomaly," where intermediate weights outperform fully converged ones.

Introduction

In modern AI, transfer learning is the standard: a model pre-trained on a massive dataset (like ImageNet) is fine-tuned for a specific, smaller task. Usually, engineers select the final, highest-performing checkpoint from the source task. This paper identifies a major gap in this practice: the best source model is not always the best transfer model. In some cases, a model still learning the source task transfers better than one that has fully converged. Picking the wrong checkpoint can even lead to results inferior to training from scratch. To solve this without the massive computational cost of fine-tuning every checkpoint, we propose a predictive metric to identify optimal transferability.

Methodology: Fréchet Pre-train Distance

The authors propose the Fréchet Pre-train Distance (FPD), inspired by the Fréchet Inception Distance (FID) used to evaluate GANs. The intuition is that if a pre-trained network perceives the source and target datasets as having similar statistical distributions in its internal feature space, it is likely to transfer well.

  1. Feature Extraction: For a given checkpoint, source and target data are passed through the network to obtain feature representations.
  2. Statistical Modeling: These features are modeled as multivariate Gaussian distributions, calculating their respective means and covariances.
  3. Distance Calculation: The Fréchet distance between these two distributions is computed.

A lower FPD indicates higher similarity and, consequently, higher predicted transferability. The authors also explore Spectral Normalization (SN) to stabilize weights and prevent the over-parameterization that often ruins transferability in early training stages.

Experimental Setup

The study utilized standard computer vision benchmarks:

  • Source Datasets: CIFAR-100 and SVHN (Street View House Numbers).
  • Target Datasets: CIFAR-10 and MNIST.
  • Architectures: AlexNet, VGG-16, and various ResNet flavors (18, 50, 101).

To simulate realistic "small-data" scenarios, the researchers used reduced versions of target sets (e.g., 10% of CIFAR-10 or 1% of MNIST).

Transfer Dynamics and the AlexNet Anomaly

Experimental results reveal complex transfer dynamics across different architectures. A notable finding is the "AlexNet Anomaly": in early training stages, AlexNet's transfer performance often dips significantly below the baseline of training from scratch. This is attributed to its massive, over-parameterized fully connected layers which initially learn noisy or overly specific features before stabilizing.

Transfer performance across architectures for the SVHN-to-MNIST task, illustrating the early-stage performance dip in AlexNet.

Key Results

  • The Transfer Peak: Transfer performance often peaks before the source task finishes training. As the model continues to train, it may overfit the source domain, causing transfer performance to drop. For example, in ResNet-18, a clear peak is observed followed by a decline in transfer accuracy.
  • Predictive Power of FPD: FPD successfully identified optimal checkpoints across different architectures and learning rates. On the CIFAR100-to-CIFAR10 task using AlexNet, transfer accuracy dropped by as much as 5 percentage points if the final checkpoint was used instead of the optimal early checkpoint identified by FPD.
Transfer performance trends for CIFAR-100 to CIFAR-10, highlighting the 'Transfer Peak' in ResNet-18 and the performance drop in late-stage training.

Conclusion

FPD provides a quantifiable method to select pre-trained checkpoints, saving significant computational resources. While currently limited by the Gaussian assumption and a focus on image classification, it offers a robust framework for understanding and optimizing transfer learning. Future work will investigate its applicability to complex tasks like object detection and non-visual domains like Natural Language Processing.

Summary updated

Abstract

Transfer learning has become the de facto practice to reuse a deep neural network (DNN) that is pre-trained with abundant training data in a source task to improve the model training on target tasks with smaller-scale training data. In this paper, we first investigate the correlation between the DNN's pre-training performance in the source task and their transfer results in the downstream tasks. We find that high performance of a pre-trained model does not necessarily imply high transferability. We then propose a metric, named Fréchet Pre-train Distance, to estimate the transferability of a deep neural network. By applying the proposed Fréchet Pre-train Distance, we are able to identify the optimal pre-trained checkpoint, and then achieve high transferability on downstream tasks. Finally, we investigate several factors impacting DNN's transferability including normalization, different networks and learning rates. The results consistently support our conclusions.
BibTeX
@inproceedings{ding2021analyzing,
  author = {Yifan Ding and Liqiang Wang and Boqing Gong},
  title = {{Analyzing Deep Neural Network's Transferability via Fréchet Distance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00398},
}
Figure