Towards accurate knowledge transfer via target-awareness representation disentanglement

Xingjian Li, Di Hu, Xuhong Li, Haoyi Xiong, Cheng-Zhong Xu, Dejing Dou

Abstract

Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (\(\textrm{TRED}\)), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Two alternative approaches, maximizing Maximum Mean Discrepancy (Max-MMD) and minimizing mutual information (Min-MI) are introduced to achieve the desired disentanglement. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average. \(\textrm{TRED}\) also outperforms related state-of-the-art transfer learning regularizers such as \(\mathrm {L^2\text {-}SP}\), \(\textrm{AT}\), \(\textrm{DELTA}\), and \(\textrm{BSS}\). Moreover, our solution is compatible with different choices of disentangling strategies. While the combination of Max-MMD and Min-MI typically achieves higher accuracy, only using Max-MMD can be a preferred choice in applications with low resource budgets.
BibTeX
@article{li2024towards,
  author = {Xingjian Li and Di Hu and Xuhong Li and Haoyi Xiong and Cheng-Zhong Xu and Dejing Dou},
  title = {{Towards accurate knowledge transfer via target-awareness representation disentanglement}},
  journal = {Machine Learning},
  year = {2024},
  doi = {10.1007/s10994-023-06381-2},
}
Figure