A Survey on Deep Transfer Learning

Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, Chunfang Liu

Introduction: The Data Dependence Problem

Deep learning has revolutionized fields from computer vision to natural language processing. However, its success is heavily predicated on the availability of massive, well-annotated datasets. This "Data Dependence" is a significant bottleneck in specialized domains like medical imaging or robotics, where data collection is expensive, rare, or physically difficult.

Traditional machine learning operates under the assumption that training and test data are independent and identically distributed (i.i.d.). Transfer learning relaxes this constraint, enabling models to leverage knowledge from a "Source Domain" with abundant data to improve performance in a "Target Domain" where data is scarce. This survey explores how deep neural networks—with their non-linear mapping capabilities—supercharge this process.

The general framework of transfer learning, showing knowledge transfer from a source to a target domain.

A Taxonomy of Deep Transfer Learning (DTL)

The authors propose a novel taxonomy, classifying DTL into four primary technical categories based on how they utilize deep neural networks:

1. Instances-based DTL

This approach assumes that while the source and target domains differ, specific instances from the source can be reused if weighted correctly. Algorithms like TrAdaBoost are employed to re-weight source data, filtering out samples that are too dissimilar to the target task to prevent "negative transfer."

2. Mapping-based DTL

Mapping-based approaches transform instances from both domains into a shared, latent data space. By minimizing the distance between these distributions—often using metrics like Maximum Mean Discrepancy (MMD)—the model can learn features that are effective across both domains.

Mapping-based DTL transforms source and target data into a new, shared data space to align their distributions.

3. Network-based DTL

Commonly known as "fine-tuning," this is the most prevalent approach in industry. It reuses partial network structures and parameters pre-trained on massive datasets (like ImageNet). The initial layers act as versatile feature extractors for universal patterns (edges, textures), while the final layers are specialized for the target task.

4. Adversarial-based DTL

Inspired by Generative Adversarial Networks (GANs), this method uses an adversarial layer to find features that are discriminative for the main task but indiscriminate between the source and target domains. This forces the network to ignore domain-specific "noise" and focus on universal, robust features.

Key Insights and Results

A critical observation in DTL is the near-linear relationship between model scale and data requirements. DTL provides the only viable path for applying state-of-the-art large-scale models to niche problems. The survey highlights several critical findings from the literature:

  • Architectural Transferability: Research indicates that architectures like ResNet, VGG, and Inception are particularly "transfer-friendly" due to their modular designs.
  • Mathematical Alignment: Techniques utilizing Wasserstein distance or Joint Maximum Mean Discrepancy (JMMD) have shown superior performance in aligning complex, high-dimensional data distributions that simple linear methods cannot touch.
  • Layer Versatility: Early layers of neural networks function similarly to the human brain’s iterative abstraction process, learning versatile features applicable to almost any visual task.

Future Directions and Challenges

Despite its promise, DTL faces several hurdles identified by the authors:

  1. Negative Transfer: If the source and target domains are too disparate, transferring knowledge can actually degrade performance. Developing robust metrics for "transferability" remains a priority.
  2. Unsupervised DTL: Moving beyond supervised learning to transfer knowledge into domains with no labels is the next major research frontier.
  3. Interdisciplinary Foundations: The authors call for "stronger physical support" for DTL, suggesting that collaboration with neuroscientists and physicists is needed to understand the fundamental nature of "knowledge" within neural architectures.

Summary updated

BibTeX
@inproceedings{tan2018survey,
  author = {Chuanqi Tan and Fuchun Sun and Tao Kong and Wenchang Zhang and Chao Yang and Chunfang Liu},
  title = {{A Survey on Deep Transfer Learning}},
  booktitle = {ICANN},
  year = {2018},
  doi = {10.1007/978-3-030-01424-7\_27},
}
Figure