Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement

Xingjian Li, Di Hu, Xuhong Li, Haoyi Xiong, Zhi Ye, Zhipeng Wang, Chengzhong Xu, Dejing Dou

Abstract

Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigms given a limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists an obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Specifically, we design two alternative methods, maximizing the Maximum Mean Discrepancy (Max-MMD) and minimizing the mutual information (Min-MI), for the representation disentanglement. Experiments on various real-world datasets show that our method stably improves the standard fine-tuning by more than 2% on average. TRED also outperforms related state-of-the-art transfer learning regularizers such as L2-SP, AT, DELTA, and BSS.

Introduction

Transfer learning is fundamental to modern computer vision, typically involving the fine-tuning of massive models pre-trained on datasets like ImageNet for specific downstream tasks. While effective, this process is susceptible to "negative transfer," where irrelevant knowledge from the source domain acts as noise, or "catastrophic forgetting," where useful general knowledge is lost too quickly.

Standard approaches often utilize the Starting Point As Reference (SPAR) method, regularizing the target model to remain close to the pre-trained weights or features. However, because pre-trained models are generalists, they contain features (e.g., specific textures or background objects) that may be irrelevant to the target task. Indiscriminately forcing a target model to mimic these features can degrade performance.

Methodology: The TRED Framework

The proposed Target-awareness REpresentation Disentanglement (TRED) framework addresses these issues through a two-step process designed to "purify" the knowledge being transferred.

The TRED framework: Step 1 involves representation disentanglement using a source model, while Step 2 applies the disentangled positive features as a regularizer during target model fine-tuning.

Step 1: Representation Disentanglement

Before fine-tuning the primary model, a lightweight "disentangler" module is trained. This module splits the original feature map into two distinct components: a positive part (task-relevant) and a negative part (task-irrelevant). To ensure a meaningful split, the authors employ:

  • Reconstruction Loss: Ensures the two parts can be recombined to form the original feature map.
  • Classification Loss: Ensures the positive part retains sufficient information to predict target labels.
  • Independence Constraints: To force the parts to be distinct, the authors propose two methods: Max-MMD (Maximum Mean Discrepancy), which forces the parts to focus on different spatial regions, and Min-MI (Minimum Mutual Information), which ensures semantic independence between channels.

Step 2: Target-aware Fine-tuning

Once disentangled, the disentangler is frozen. During the fine-tuning of the target model, only the positive features serve as the reference for regularization. This ensures the model is only "reminded" of knowledge that actively aids the new task, mitigating the impact of irrelevant source data.

Experimental Evaluation

The researchers evaluated TRED across seven diverse datasets, including Stanford Dogs, CUB-200-2011 (birds), and MIT Indoor-67.

Key Results

  • Superior Accuracy: TRED consistently outperformed standard fine-tuning ($L^2$) and state-of-the-art regularizers like $L^2$-SP and DELTA, improving top-1 accuracy by an average of over 2%.
  • Data Efficiency: TRED demonstrated significant gains in low-data regimes. For instance, when using only 15% of the CUB-200 dataset, TRED-MI achieved 42.07% accuracy, compared to 34.98% for standard $L^2$ regularization.
  • Hyperparameter Robustness: Unlike many transfer learning methods that are sensitive to regularization strength ($\alpha$), TRED remains stable even as $\alpha$ increases, as the reference signal is inherently cleaner.

Visualization and Analysis

To confirm the effectiveness of the disentanglement, the authors utilized t-SNE visualizations and attention maps. These tools provide visual evidence that the disentangler successfully separates the "object of interest" from background noise.

t-SNE visualizations for Flower-102 and Indoor-67 datasets, comparing the original feature distributions with the disentangled representations.

The t-SNE plots demonstrate that the disentangled features often result in more distinct clusters compared to the original features, suggesting that the purification process enhances the discriminative power of the representations for the target task.

Conclusion

TRED offers a robust solution to the problem of negative transfer by explicitly separating relevant and irrelevant knowledge before fine-tuning. By purifying the reference signal used in SPAR regularization, TRED achieves superior accuracy and data efficiency across a wide range of visual tasks. Future research may extend this disentanglement approach to non-visual domains such as Natural Language Processing or explore its utility in zero-shot learning scenarios.

Summary updated

Abstract

Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Specifically, we design two alternative methods, maximizing the Maximum Mean Discrepancy (Max-MMD) and minimizing the mutual information (Min-MI), for the representation disentanglement. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average. TRED also outperforms related state-of-the-art transfer learning regularizers such as L2-SP, AT, DELTA, and BSS.
BibTeX
@article{li2024towards,
  author = {Xingjian Li and Di Hu and Xuhong Li and Haoyi Xiong and Zhi Ye and Zhipeng Wang and Chengzhong Xu and Dejing Dou},
  title = {{Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement}},
  journal = {Machine Learning},
  year = {2024},
  doi = {10.1007/s10994-023-06381-2},
}
Figure