P2L: Predicting Transfer Learning for Images and Semantic Relations

Bishwaranjan Bhattacharjee, John R. Kender, Matthew Hill, Parijat Dube, Siyu Huo, Michael R. Glass, Brian M. Belgodere, Sharath Pankanti, Noel Codella, Patrick Watson

Abstract

We describe an efficient method to accurately estimate the effectiveness of a previously trained deep learning model for use in a new learning task. We use this method, "Predict To Learn" (P2L), to predict the most likely "source" dataset to produce effective transfer for training on a "target" dataset. We validate our approach extensively across 21 tasks, including image classification tasks and semantic relationship prediction tasks in the linguistic domain. The P2L approach selects the best transfer learning model on 62% of the tasks, compared with a baseline of 48% of cases when using a heuristic of selecting the largest source dataset and 52% of cases when using a distance measure between source and target datasets. Further, our work results in an 8% reduction in error rate. Finally, we also show that a model trained from merging multiple source model datasets does not necessarily result in improved transfer learning. This suggests that performance of the target model depends upon the relative composition of the source dataset as well as their absolute scale, as measured by our novel method we term 'P2L'.

Introduction

Transfer learning has become a cornerstone of modern artificial intelligence, enabling practitioners to leverage models pre-trained on massive "source" datasets for specialized "target" tasks. However, selecting the optimal source model remains a significant challenge. A poor choice can lead to negative transfer, where pre-trained weights perform worse than random initialization.

Currently, many practitioners rely on the "biggest is best" heuristic, choosing models trained on the largest available datasets (e.g., ImageNet-21K). As datasets become more specialized and computationally expensive to train, this brute-force approach is increasingly inefficient. We address this by asking: How can we accurately predict transfer learning effectiveness prior to training, using minimal compute?

Methods: The P2L Estimator

We introduce Predict To Learn (P2L), a lightweight algorithm designed to identify the best pre-trained source model without expensive trial-and-error training. P2L utilizes a metric we call Embedding Divergence, calculated in three distinct steps:

  1. Feature Extraction: A single "reference model" (such as VGG16) performs a forward pass on both source and target datasets to generate representative feature vectors or embeddings.
  2. Distance Calculation: We measure the "distance" ($D$) between target and source features using metrics like KL-Divergence or Euclidean distance.
  3. The P2L Estimator: We combine source dataset size and feature distance into a single score: $$E(t_i, s_j) = z(\log(|s_j|)) + k \cdot z(D(t_i, s_j))$$ Here, $|s_j|$ represents source size, $D$ is the distance, and $k$ is a learned parameter balancing the two factors.
The P2L feature extraction pipeline. A reference CNN model generates embeddings which are then averaged into a single representative feature vector for the dataset.

This "offline" method is remarkably fast. For instance, predicting the best source for the Describable Textures Dataset (DTD) took only 65 seconds, compared to 12 hours for a full training run.

Key Insights and Results

Our experiments revealed several critical findings regarding transfer learning dynamics:

  • Size vs. Similarity: While larger source datasets generally improve performance, the benefit is logarithmic. For niche target tasks, the domain similarity of the source becomes the dominant predictor of success.
  • The "Merged Dataset" Myth: Merging multiple source datasets into a single "super-dataset" does not guarantee improved transfer. If the merge increases divergence relative to the target task, performance can degrade.
  • Predictive Accuracy: P2L selected the optimal transfer model in 62% of tasks, outperforming the "largest source" (48%) and "least distance" (52%) heuristics. This led to an overall 8% reduction in error rate.
Spearman correlation between the P2L estimator and ground-truth performance for semantic relation tasks, demonstrating high predictive reliability.

The Spearman correlation between P2L's predictions and actual performance was high: 0.707 for image tasks and 0.763 for semantic relations, confirming the estimator's reliability as a proxy for real-world training outcomes.

Conclusion

P2L offers a practical, efficient solution for model selection in transfer learning. By requiring only a single forward pass through a reference model, it is highly suitable for cloud-based environments where users need optimized models for diverse datasets quickly. Future work will explore multi-modal sources and finer-grained feature identification to further refine selection accuracy.

Summary updated

Abstract

We describe an efficient method to accurately estimate the effectiveness of a previously trained deep learning model for use in a new learning task. We use this method, "Predict To Learn" (P2L), to predict the most likely "source" dataset to produce effective transfer for training on a "target" dataset. We validate our approach extensively across 21 tasks, including image classification tasks and semantic relationship prediction tasks in the linguistic domain. The P2L approach selects the best transfer learning model on 62% of the tasks,compared with a baseline of 48% of cases when using a heuristic of selecting the largest source dataset and 52% of cases when using a distance measure between source and target datasets. Further, our work results in an 8% reduction in error rate. Finally, we also show that a model trained from merging multiple source model datasets does not necessarily result in improved transfer learning. This suggests that performance of the target model depends upon the relative composition of the source dataset as well as their absolute scale, as measured by our novel method we term 'P2L'
BibTeX
@inproceedings{bhattacharjee2020p,
  author = {Bishwaranjan Bhattacharjee and John R. Kender and Matthew Hill and Parijat Dube and Siyu Huo and Michael R. Glass and Brian M. Belgodere and Sharath Pankanti and Noel Codella and Patrick Watson},
  title = {{P2L: Predicting Transfer Learning for Images and Semantic Relations}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=lreQYTBK2Du},
}
Figure