Learning What and Where to Transfer

Yunhun Jang, Hankook Lee, Sung Ju Hwang, Jinwoo Shin

TL;DR

Transfer learning is often a "black art" requiring researchers to manually decide which layers and features to transfer between models. This paper introduces a meta-learning framework that automatically learns what (which feature channels) and where (which layer pairs) to transfer, significantly outperforming hand-crafted baselines—especially when the source and target architectures are different or data is scarce.

What Problem Does This Paper Address?

Transfer learning is the standard solution for training deep networks on small datasets: you take a model pre-trained on a massive dataset (like ImageNet) and "fine-tune" it for a new task. However, this process becomes incredibly difficult when the source and target models have different architectures (e.g., transferring from a ResNet to a VGG) or when the tasks are semantically distant.

Existing methods like "FitNet" or "Attention Transfer" require humans to manually pair up layers between the two networks. This is tedious, often sub-optimal, and ignores the fact that some feature channels are more relevant to the target task than others. The authors aim to replace this manual "trial-and-error" with an automated, learnable process.

Methods

The authors propose a method called L2T-ww (Learning What and Where to transfer). The core innovation lies in two "meta-networks" ($f$ and $g$) that act as gatekeepers for the transfer process:

Comparison of previous transfer methods and the proposed L2T-ww method
  1. Where to Transfer (Layer Matching): A meta-network ($g$) learns a weight ($\lambda$) for every possible pair of layers between the source and target. This allows the system to discover non-obvious connections, such as a middle layer in the source being highly relevant to a late layer in the target.
  2. What to Transfer (Channel Weighting): A second meta-network ($f$) assigns weights ($w$) to individual feature channels. This ensures the target model mimics the "useful" features (like object shapes) while ignoring irrelevant ones (like background noise or source-task-specific artifacts).
  3. 3-Stage Meta-Optimization: To train these meta-networks efficiently, the authors use a bilevel optimization scheme. Instead of a slow standard loop, they use a three-stage approach:
  4. Stage 1: Update the target model to minimize the transfer loss (mimicking the source).
  5. Stage 2: Perform a one-step adaptation toward the actual target labels.
  6. Stage 3: Update the meta-networks based on how well that one-step adaptation performed.

Datasets

The researchers evaluated their method across two scales:

  • Small Scale (32x32): Source task was TinyImageNet; target tasks included CIFAR-10, CIFAR-100, and STL-10.
  • Large Scale (224x224): Source task was ImageNet; target tasks focused on specialized, fine-grained recognition: CUB200 (birds), MIT67 (indoor scenes), Stanford 40 Actions, and Stanford Dogs.

Key Insights and Ideas

  • Automated Architecture Agnosticism: By learning the mapping between layers, the method handles heterogeneous architectures (e.g., ResNet32 to VGG9) without any human intervention.
  • Task-Specific Saliency: Visualizations show that the learned weights ($w$) naturally focus on the actual objects in an image. While standard feature matching might try to transfer everything (including the grass behind a bird), L2T-ww learns to ignore the background.
  • The "General-to-Specific" Gradient: The meta-networks discovered that lower layers in the source are generally useful for all samples, while higher-level features need to be transferred more selectively depending on the specific input image.
  • Efficiency in Data Scarcity: The method shines brightest when the target dataset is tiny. With only 50 samples per class, L2T-ww achieved accuracy comparable to baseline methods using 250 samples per class.

Key Results

  • State-of-the-Art Performance: On the CUB200 (birds) dataset, L2T-ww achieved 65.05% accuracy, a massive jump over the best baseline (58.90%).
  • Multi-Source Transfer: The method effectively aggregates knowledge from multiple source models simultaneously. When transferring from both a ResNet and a VGG to a new task, L2T-ww automatically balanced the contributions from each, outperforming single-source transfer.
  • Robustness: Across all tested datasets (CIFAR, Dogs, Actions), the automated "all-to-all" matching scheme consistently beat hand-crafted "one-to-one" or "single-layer" matching configurations.

Limitations and Future Work

  • Computational Overhead: While the 3-stage optimization is faster than standard meta-learning, training meta-networks still adds a computational cost during the training phase compared to simple fine-tuning.
  • Search Space: For very deep networks, the "all-to-all" candidate pair set ($C$) could become quite large, potentially requiring more memory or a more constrained search space.
  • Beyond Vision: The paper focuses exclusively on Convolutional Neural Networks (CNNs) for image classification. Applying this "what and where" logic to Transformers or Natural Language Processing (NLP) tasks remains an open and promising direction.

Summary updated

Abstract

As the application of deep learning has expanded to real-world problems with insufficient volume of training data, transfer learning recently has gained much attention as means of improving the per...
BibTeX
@inproceedings{jang2019learning,
  author = {Yunhun Jang and Hankook Lee and Sung Ju Hwang and Jinwoo Shin},
  title = {{Learning What and Where to Transfer}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=ryNMAjWu-S},
}
Figure