Towards Making Deep Transfer Learning Never Hurt

Ruosi Wan, Haoyi Xiong, Xingjian Li, Zhanxing Zhu, Jun Huan

Abstract

Transfer learning is a cornerstone of modern deep learning, yet it remains susceptible to "negative transfer"—a phenomenon where pre-trained knowledge degrades performance on a target task. This paper introduces DTNH (Deep Transfer learning Never Hurt), a gradient-based optimization strategy designed to eliminate conflicts between empirical loss minimization and regularization. By ensuring that regularization gradients do not oppose the task-specific descent direction, DTNH guarantees that transferring knowledge consistently improves model accuracy.

Introduction

The standard paradigm in deep learning involves fine-tuning models pre-trained on massive datasets like ImageNet. To preserve pre-trained features, researchers employ regularizers such as $L^2$-SP or Knowledge Distillation. However, when source and target tasks are poorly aligned (e.g., identifying dog breeds vs. classifying furniture), these regularizers can introduce conflicting gradients. This "tug-of-war" between the task loss and the regularizer leads to negative transfer, where the model performs worse than if trained from scratch.

The DTNH Optimization Strategy

The authors propose DTNH to manage the interaction between the empirical loss ($J$) and the regularizer ($\Omega$). Rather than simply summing their gradients, DTNH employs a conflict-aware update rule:

  1. Gradient Estimation: Stochastic gradients for the empirical loss ($\nabla J_t$) and the regularizer ($\nabla \Omega_t$) are calculated independently.
  2. Conflict Detection: The algorithm measures the angle between these vectors. If the angle is obtuse ($> 90^\circ$), the regularizer is actively hindering the minimization of the empirical loss.
  3. Orthogonal Decomposition and Correction: In cases of conflict, DTNH decomposes the regularizer's gradient into parallel and orthogonal components relative to the loss gradient. It then "truncates" the conflicting parallel component, using only the orthogonal component for the update.
Comparison of standard gradient aggregation versus the DTNH orthogonal decomposition strategy.

This ensures the final update direction always reduces the empirical loss while incorporating as much non-conflicting "knowledge" from the source model as possible.

Experimental Setup

DTNH was validated across 15 source-target task pairs using ResNet-18. Source datasets included ImageNet, Places 365, and Stanford Dogs 120, while target tasks spanned Caltech 256, MIT Indoors 67, Flowers 102, and CIFAR-10. These pairs were selected to test both synergistic and antagonistic transfers, specifically looking for instances of negative transfer.

Key Results and Insights

The core insight of DTNH is that knowledge transfer should be treated as a secondary constraint that must not violate the primary goal of data fitting. By using orthogonal projection, the model avoids the "narrow cone" of local minima often caused by rigid regularization.

Empirical and testing loss curves comparing DTNH with standard L2-SP regularization across training epochs.

Empirical results demonstrate:

  • Consistent Gains: DTNH improved performance across all 15 pairs, with accuracy increases between 0.1% and 7%.
  • Mitigating Negative Transfer: In scenarios where standard transfer learning failed (e.g., Places 365 $\to$ Flowers 102), DTNH successfully "rescued" the model, outperforming simple fine-tuning.
  • Superior Convergence: As shown in the loss curves, DTNH reaches lower empirical loss values faster than standard methods by eliminating gradient interference.

Conclusion and Future Work

DTNH serves as a robust meta-strategy that can be integrated with existing transfer learning methods to prevent performance degradation. While it introduces some computational overhead due to dual gradient calculations, the reliability it offers makes it a valuable tool for cross-domain applications. Future research may explore adaptive thresholding for the conflict detection angle to further optimize the balance between regularization and task-specific learning.

Summary updated

BibTeX
@inproceedings{wan2019towards,
  author = {Ruosi Wan and Haoyi Xiong and Xingjian Li and Zhanxing Zhu and Jun Huan},
  title = {{Towards Making Deep Transfer Learning Never Hurt}},
  booktitle = {ICDM},
  year = {2019},
  doi = {10.1109/icdm.2019.00068},
}
Figure