Abstract
Transfer learning is a cornerstone of modern deep learning, yet it remains susceptible to "negative transfer"—a phenomenon where pre-trained knowledge degrades performance on a target task. This paper introduces DTNH (Deep Transfer learning Never Hurt), a gradient-based optimization strategy designed to eliminate conflicts between empirical loss minimization and regularization. By ensuring that regularization gradients do not oppose the task-specific descent direction, DTNH guarantees that transferring knowledge consistently improves model accuracy.
Introduction
The standard paradigm in deep learning involves fine-tuning models pre-trained on massive datasets like ImageNet. To preserve pre-trained features, researchers employ regularizers such as $L^2$-SP or Knowledge Distillation. However, when source and target tasks are poorly aligned (e.g., identifying dog breeds vs. classifying furniture), these regularizers can introduce conflicting gradients. This "tug-of-war" between the task loss and the regularizer leads to negative transfer, where the model performs worse than if trained from scratch.
The DTNH Optimization Strategy
The authors propose DTNH to manage the interaction between the empirical loss ($J$) and the regularizer ($\Omega$). Rather than simply summing their gradients, DTNH employs a conflict-aware update rule:
- Gradient Estimation: Stochastic gradients for the empirical loss ($\nabla J_t$) and the regularizer ($\nabla \Omega_t$) are calculated independently.
- Conflict Detection: The algorithm measures the angle between these vectors. If the angle is obtuse ($> 90^\circ$), the regularizer is actively hindering the minimization of the empirical loss.
- Orthogonal Decomposition and Correction: In cases of conflict, DTNH decomposes the regularizer's gradient into parallel and orthogonal components relative to the loss gradient. It then "truncates" the conflicting parallel component, using only the orthogonal component for the update.
This ensures the final update direction always reduces the empirical loss while incorporating as much non-conflicting "knowledge" from the source model as possible.
Experimental Setup
DTNH was validated across 15 source-target task pairs using ResNet-18. Source datasets included ImageNet, Places 365, and Stanford Dogs 120, while target tasks spanned Caltech 256, MIT Indoors 67, Flowers 102, and CIFAR-10. These pairs were selected to test both synergistic and antagonistic transfers, specifically looking for instances of negative transfer.
Key Results and Insights
The core insight of DTNH is that knowledge transfer should be treated as a secondary constraint that must not violate the primary goal of data fitting. By using orthogonal projection, the model avoids the "narrow cone" of local minima often caused by rigid regularization.
Empirical results demonstrate:
- Consistent Gains: DTNH improved performance across all 15 pairs, with accuracy increases between 0.1% and 7%.
- Mitigating Negative Transfer: In scenarios where standard transfer learning failed (e.g., Places 365 $\to$ Flowers 102), DTNH successfully "rescued" the model, outperforming simple fine-tuning.
- Superior Convergence: As shown in the loss curves, DTNH reaches lower empirical loss values faster than standard methods by eliminating gradient interference.
Conclusion and Future Work
DTNH serves as a robust meta-strategy that can be integrated with existing transfer learning methods to prevent performance degradation. While it introduces some computational overhead due to dual gradient calculations, the reliability it offers makes it a valuable tool for cross-domain applications. Future research may explore adaptive thresholding for the conflict detection angle to further optimize the balance between regularization and task-specific learning.
Summary updated