Abstract
We study a transfer learning framework where source and target datasets are heterogeneous in both feature and label spaces. Specifically, we do not assume explicit relations between source and target tasks a priori, and thus it is crucial to determine what and what not to transfer from source knowledge. Towards this goal, we define a new heterogeneous transfer learning approach that (1) selects and attends to an optimized subset of source samples to transfer knowledge from, and (2) builds a unified transfer network that learns from both source and target knowledge. This method, termed "Attentional Heterogeneous Transfer", along with a newly proposed unsupervised transfer loss, improve upon the previous state-of-the-art approaches on extensive simulations as well as a challenging hetero-lingual text classification task.
Introduction
Transfer learning has revolutionized machine learning by allowing models to leverage knowledge from a source domain to improve performance on a target domain. However, most existing methods assume some degree of commonality, such as shared features or labels. In many real-world scenarios, we encounter "Completely Heterogeneous Transfer Learning" (CHTL), where source and target datasets share neither features nor labels. For instance, classifying German medical documents using English news data involves different languages and different categories. This paper addresses two primary challenges in CHTL: negative transfer, where irrelevant source data degrades performance, and data scarcity in the target domain.
Methodology
The authors propose "Attentional Heterogeneous Transfer," a framework designed to selectively transfer knowledge while maintaining robustness. The architecture consists of three key components:
- Shared Label Space: To bridge disparate label sets, labels are mapped into a common distributed space using word embeddings (e.g., Word2Vec). This allows the model to find semantic relationships between labels like "Politics" and "Government" across different languages, acting as semantic anchors.
- Attentional Transfer: Rather than a blanket transfer, the model clusters source data and learns an attention weight for each cluster. This mechanism allows the network to focus on relevant source samples and ignore those that might cause negative transfer.
- Denoising Auto-encoders (AE): To handle limited labeled target data, an unsupervised reconstruction loss is added. The model learns to reconstruct input features ($X_S, X_T$) from the latent space ($X_C$), utilizing unlabeled data to stabilize deep neural network training.
The network employs transformations $g$ and $h$ to map source and target features into a joint subspace $X_C$, followed by a shared function $f$ that maps this subspace to the joint embedded label space $Y$. This structure allows for deeper, more robust networks than previous linear CHTL methods.
Experimental Setup
The framework was evaluated on synthetic datasets with controllable levels of heterogeneity and real-world hetero-lingual text classification tasks. Real-world datasets included RCV-1 and 20 Newsgroups as source knowledge, with target tasks from Reuters Multilingual (French, Spanish, German, Italian) and R8. The hetero-lingual task specifically tested transferring knowledge from English sources to non-English targets with different category sets.
Results and Discussion
The proposed method (CHTL:ATT+AE) consistently outperformed state-of-the-art baselines, including Zero-Shot Learning (ZSL). In the English-to-German classification task, the model achieved 58.4% accuracy, significantly higher than the 51.1% achieved by ZSL.
A key finding was the model's robustness to heterogeneity. As shown in synthetic tests, while standard transfer methods fail when label spaces diverge (increasing $\sigma_{label}$), the attentional mechanism maintains stability by filtering out irrelevant source information. The model effectively performs instance-level selection, determining exactly what to transfer. Furthermore, the authors identified an optimal cluster size ($K=40$) that balances computational efficiency with the granularity of selective transfer.
Conclusion and Future Work
Attentional Heterogeneous Transfer effectively addresses the "what to transfer" problem in CHTL. By combining semantic label anchoring, selective attention, and unsupervised regularization, the model avoids negative transfer and performs well even with scarce labeled data. While successful, the method currently relies on the quality of label embeddings and pre-processing clustering steps. Future work may explore end-to-end learnable clustering and cross-modal transfer tasks, such as image-to-text transfer.
Summary updated