Learning What and Where to Transfer

Yunhun Jang, Hankook Lee, Sung Ju Hwang, Jinwoo Shin

Abstract

As the application of deep learning has expanded to real-world problems with insufficient volume of training data, transfer learning recently has gained much attention as means of improving the performance in such small-data regime. However, when existing methods are applied between heterogeneous architectures and tasks, it becomes more important to manage their detailed configurations and often requires exhaustive tuning on them for the desired performance. To address the issue, we propose a novel transfer learning approach based on meta-learning that can automatically learn what knowledge to transfer from the source network to where in the target network. Given source and target networks, we propose an efficient training scheme to learn meta-networks that decide (a) which pairs of layers between the source and target networks should be matched for knowledge transfer and (b) which features and how much knowledge from each feature should be transferred. We validate our meta-transfer approach against recent transfer learning methods on various datasets and network architectures, on which our automated scheme significantly outperforms the prior baselines that find "what and where to transfer" in a hand-crafted manner.

Introduction

Standard transfer learning, such as fine-tuning a pre-trained ResNet, is highly effective when source and target tasks share significant similarities. However, its efficacy diminishes in two critical scenarios: when architectures are heterogeneous (e.g., transferring from a deep ResNet to a shallow VGG) and when tasks are distant (e.g., moving from general ImageNet categories to specialized fine-grained classification). In these cases, not all source features are beneficial; some may even lead to "negative transfer."

Existing solutions often require researchers to manually design "matching schemes"—deciding which specific layers in the source should correspond to those in the target. This trial-and-error process is labor-intensive and often sub-optimal. To overcome this, we introduce L2T-ww (Learning What and Where to transfer), a meta-learning framework that automates the selection of feature channels and layer-to-layer connections.

The L2T-ww Framework

The core of L2T-ww consists of two meta-networks designed to optimize the transfer process:

  1. Weighted Feature Matching ("What"): A meta-network ($f_\phi$) evaluates source features and assigns weights to individual channels. This allows the model to prioritize task-relevant features (like the specific shape of a bird's beak) while ignoring irrelevant background noise.
  2. Layer Association ("Where"): A second meta-network ($g_\phi$) determines the connection strength between any pair of layers from the source and target networks. This enables the discovery of non-obvious relationships, such as a source layer being highly relevant to a much later target layer.
Comparison of previous transfer learning methods versus the proposed L2T-ww approach.

To train these meta-networks, we employ a three-stage bilevel optimization scheme. The "inner loop" trains the target model to mimic the source based on current meta-weights, while the "outer loop" updates the meta-networks based on the target model's actual performance on the final task.

Experimental Evaluation

We evaluated L2T-ww across various scales and architectures:

  • Small-scale: Source task TinyImageNet ($32 \times 32$); target tasks CIFAR-10, CIFAR-100, and STL-10.
  • Large-scale: Source task ImageNet ($224 \times 224$); target tasks included fine-grained recognition datasets like CUB200 (Birds), MIT67 (Indoor scenes), Stanford 40 Actions, and Stanford Dogs.
  • Architectures: Transfers were tested between ResNet-34/32 and VGG-9/ResNet-18, demonstrating cross-family compatibility.

Results and Insights

L2T-ww consistently outperformed standard baselines. On the CUB200 dataset, it achieved 65.05% accuracy, a substantial improvement over the 58.90% achieved by Attention Transfer.

A key finding is that automated "Where" matching is non-trivial. As shown in Figure 6d, the learned configurations often involve complex, many-to-one mappings rather than simple sequential pairings. This flexibility allows the model to aggregate knowledge from multiple source depths into a single target layer.

Different layer matching strategies, including the flexible 'Learned matching' configuration.

Furthermore, L2T-ww proves exceptionally robust in small-data regimes. With only 50 samples per class, it achieves accuracy comparable to other methods using five times as much data. Visualization via saliency maps further confirms that the method naturally learns to focus on salient objects while ignoring backgrounds, a direct result of the "What" meta-network's weighting.

Conclusion

L2T-ww addresses a major bottleneck in transfer learning by automating the configuration of knowledge transfer. While the bilevel training introduces some computational overhead, the resulting models maintain standard inference speeds and significantly higher accuracy, especially in challenging heterogeneous or low-data scenarios. Future work may explore extending this framework to non-vision tasks and optimizing the search space for extremely deep architectures.

Summary updated

Abstract

As the application of deep learning has expanded to real-world problems with insufficient volume of training data, transfer learning recently has gained much attention as means of improving the performance in such small-data regime. However, when existing methods are applied between heterogeneous architectures and tasks, it becomes more important to manage their detailed configurations and often requires exhaustive tuning on them for the desired performance. To address the issue, we propose a novel transfer learning approach based on meta-learning that can automatically learn what knowledge to transfer from the source network to where in the target network. Given source and target networks, we propose an efficient training scheme to learn meta-networks that decide (a) which pairs of layers between the source and target networks should be matched for knowledge transfer and (b) which features and how much knowledge from each feature should be transferred. We validate our meta-transfer approach against recent transfer learning methods on various datasets and network architectures, on which our automated scheme significantly outperforms the prior baselines that find "what and where to transfer" in a hand-crafted manner.
BibTeX
@inproceedings{jang2019learning,
  author = {Yunhun Jang and Hankook Lee and Sung Ju Hwang and Jinwoo Shin},
  title = {{Learning What and Where to Transfer}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=ryNMAjWu-S},
}
Figure