P2L: Predicting Transfer Learning for Images and Semantic Relations

Bishwaranjan Bhattacharjee, Noel Codella, John R. Kender, Siyu Huo, Patrick Watson, Michael R. Glass, Parijat Dube, Matthew Hill, Brian M. Belgodere

TL;DR

This paper introduces "Predict To Learn" (P2L), an efficient method to predict which pre-trained model will best facilitate transfer learning for a new task. P2L outperforms simple heuristics like choosing the largest source dataset, demonstrating that the similarity between source and target tasks is crucial for successful transfer learning, not just dataset size.

What Problem Does This Paper Address?

Deep learning models often require vast amounts of labeled data, which is scarce in many real-world scenarios. Transfer learning offers a solution by leveraging knowledge from pre-trained models on "source" tasks to improve performance on new "target" tasks. However, simply picking a pre-trained model isn't always effective; a poorly chosen source model can even hurt performance compared to training from scratch. The challenge is to efficiently predict which pre-trained model will yield the best transfer learning results before undertaking the computationally expensive process of fine-tuning. This is particularly important for cloud-based AI services that need to quickly and accurately train models for diverse user tasks.

Methods

The core of the paper is the "Predict To Learn" (P2L) method. P2L estimates the effectiveness of transfer learning by considering two key factors:

  • Source Dataset Size: The paper acknowledges that larger source datasets generally lead to better transfer, but notes this is not the sole determinant.
  • Embedding Divergence: This is the novel component. P2L quantifies the "distance" or "divergence" between the feature representations learned by a source model and the target task. This is achieved by:
  • Generating a single, summarizing feature vector for each dataset (both source and target).
  • Computing a dissimilarity measure (e.g., KL Divergence, Jensen-Shannon Distance, Chi-square, Minkowski) between these feature vectors.
  • Combining this dissimilarity measure with a logarithmic function of the source dataset size into an "embedding divergence" score.

This score, denoted as $E(t_i, s_j)$, is designed to approximate the actual performance improvement $I(t_i, s_j)$ achieved by fine-tuning a source model $s_j$ for a target task $t_i$. The parameter $k$ in the embedding divergence formula is learned empirically. P2L requires only a single forward pass of the target data through a reference model to generate its feature vector, making it computationally efficient.

Datasets

The paper evaluates P2L on two distinct domains:

  • Images:
    • Target Datasets: Caltech-UCSD Birds (CUBS), Stanford Cars, Sketches, Wikiart, Oxford Flowers, and 9 datasets from the Visual Decathlon Challenge (including Aircraft, CIFAR100, Daimler Pedestrian Classification, Describable Textures (DTD), German Traffic Sign (GTSRB), Omniglot, Street View House Number (SVHN), UCF101, VGG-Flowers).
  • Source Datasets: Places1, ImageNet1K, and 15 subsets derived from ImageNet22K (which contains hierarchical categories like person, animal, fungus, etc.).
  • Semantic Relations:
    • Target Datasets: Seven subtasks derived from the CC-DBP (Common Crawl - DBpedia) dataset, focusing on high-level relations.
  • Source Datasets: Models trained on each of the seven subtasks from CC-DBP.

Key Insights and Ideas

  • Beyond Dataset Size: The most significant insight is that simply choosing the largest source dataset is not optimal for transfer learning. P2L demonstrates that the similarity between the source and target task's feature representations is a critical factor, often more so than raw data volume.
  • Feature Space Similarity Matters: The paper highlights that understanding the "embedding divergence" – how similar the learned feature spaces are – is key to predicting successful transfer.
  • Efficiency is Key: P2L's ability to predict transfer effectiveness with a single forward pass makes it practical for real-world applications like cloud-based APIs, where exhaustive search is infeasible.
  • No Universal "Best" Source: The results suggest that the optimal source model is task-dependent, and there isn't a single model that works best across all target tasks.
  • Merging Datasets Isn't Always Better: The paper found that combining multiple source datasets into a single large one doesn't always outperform selecting a single, well-matched source, especially if the merged dataset becomes too diverse.

Key Results

  • Improved Accuracy: P2L achieved an average Top-1 accuracy of 67.22% across 21 tasks, outperforming the baseline of picking the largest source dataset (64.47%) and the least divergent dataset (64.86%).
  • Better Model Selection: P2L successfully selected the best transfer learning model in 62% of tasks, compared to 48% for the largest dataset heuristic and 52% for a distance-based heuristic.
  • High Correlation: The Spearman correlation between P2L's predictions and the ground truth transfer performance was strong: 0.707 for images and 0.763 for semantic relations.
  • Error Reduction: The paper reports an 8% reduction in error rate due to P2L.
  • Efficiency: For image tasks, P2L was able to pick the best source model within an average of 3 picks, significantly fewer than baseline methods. The prediction latency was also low (e.g., 65 seconds for a DTD dataset on a K40 GPU).

Limitations and Future Work

  • Limitations:
    • The current P2L approach primarily considers large conceptual categories. It doesn't deeply explore finer-grained hierarchical features within datasets.
  • The method currently uses only one modality (visual or semantic) in isolation.
  • Future Work:
    • Finer-grained Feature Characterization: Exploring deeper hierarchies within datasets to identify crucial sub-features for more precise source selection.
  • Multi-modal Transfer: Incorporating information from other modalities (e.g., text, audio) to improve source model selection.
  • Enriched Relationship Modeling: Developing more complex models for dataset representation and exploring richer relationships beyond simple similarity, such as dispersion statistics.
  • Extensibility: The framework is designed to be extensible to learn other compact representations, inexpensive dissimilarity functions, and more sophisticated non-linear modeling of size-distance interactions.

Summary updated

Abstract

Transfer learning enhances learning across tasks, by leveraging previously learned representations -- if they are properly chosen. We describe an efficient method to accurately estimate the appropriateness of a previously trained model for use in a new learning task. We use this measure, which we call "Predict To Learn" ("P2L"), in the two very different domains of images and semantic relations, where it predicts, from a set of "source" models, the one model most likely to produce effective transfer for training a given "target" model. We validate our approach thoroughly, by assembling a collection of candidate source models, then fine-tuning each candidate to perform each of a collection of target tasks, and finally measuring how well transfer has been enhanced. Across 95 tasks within multiple domains (images classification and semantic relations), the P2L approach was able to select the best transfer learning model on average, while the heuristic of choosing model trained with the largest data set selected the best model in only 55 cases. These results suggest that P2L captures important information in common between source and target tasks, and that this shared informational structure contributes to successful transfer learning more than simple data size.
BibTeX
@inproceedings{bhattacharjee2020p,
  author = {Bishwaranjan Bhattacharjee and Noel Codella and John R. Kender and Siyu Huo and Patrick Watson and Michael R. Glass and Parijat Dube and Matthew Hill and Brian M. Belgodere},
  title = {{P2L: Predicting Transfer Learning for Images and Semantic Relations}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=lreQYTBK2Du},
}
Figure