Heterogeneous Domain Adaptation with Positive and Unlabeled Data

Junki Mori, Ryo Furukawa, Isamu Teranishi, Jun Sakuma

TL;DR

This paper introduces a new machine learning challenge: PU-HUDA (Positive and Unlabeled Heterogeneous Unsupervised Domain Adaptation). It addresses scenarios where you want to train a model for a new "target" domain using a "source" domain that has entirely different features and—crucially—only contains positive examples. The authors propose PADA (Predictive Adversarial Domain Adaptation), a framework that uses adversarial training and a "soft-labeling" mechanism to align disparate feature spaces while accurately identifying positive samples in unlabeled data.

What Problem Does This Paper Address?

In the world of transfer learning, Unsupervised Domain Adaptation (UDA) is a common task: you have a labeled "source" dataset and want to train a model for an unlabeled "target" dataset. However, standard UDA assumes the source and target share the same features (e.g., both are images of the same resolution).

Heterogeneous UDA (HUDA) ups the ante by assuming the feature spaces are different (e.g., the source uses text descriptions while the target uses numerical metadata). This paper identifies a further, more difficult gap: PU-HUDA. In many real-world cases, the source domain only has "positive" labels.

Imagine a jewelry store (source) partnering with a bank (target) to find potential customers. The jewelry store only knows who did buy jewelry (positives); it doesn't have a list of people who definitely won't. Meanwhile, the bank has a massive database of unlabeled customers, and its data features (account balances) are totally different from the store's features (past purchase history). Naively combining existing methods fails because they try to align the entire target distribution with the positive-only source distribution, effectively "smearing" the target's positive and negative classes together and destroying the model's predictive power.

Methods

The authors propose PADA (Predictive Adversarial Domain Adaptation), which consists of three interacting components:

  1. Feature Transformer ($F$): Maps the target's unique features into the source's feature space.
  2. Classifier ($C$): Predicts which target samples are likely "positives."
  3. Discriminator ($D$): Acts as a judge, trying to distinguish between actual source positives and the "likely positives" from the target domain.

The Adversarial Loop: $F$ and $C$ work together to "fool" $D$. $C$ identifies target samples it thinks are positive, and $F$ transforms them so they look indistinguishable from source positives to the Discriminator.

Soft-Labeling ($PADA_S$): To prevent the model from collapsing during the early stages of training (when the transformer is still "clumsy"), the authors introduce a base classifier ($C_0$). This base model is trained only on "common features"—attributes shared by both domains, like basic demographics. $C_0$ provides "soft labels" (probabilistic hints) to guide the main classifier $C$ until the adversarial process stabilizes.

Datasets

Since PU-HUDA is a new task, the authors reorganized three existing datasets to simulate this environment:

  • Movielens-Netflix: Users' movie ratings where genres serve as features. One domain is treated as source (positives only), the other as target (unlabeled).
  • 20-Newsgroups: A text classification dataset where different news categories are mapped to source and target domains using tf-idf features.
  • Default of Credit Card Clients: A dataset of credit records. The authors varied the "positive ratio" (the percentage of actual positives in the unlabeled target data) from 0.5 down to 0.1 to test the model's robustness.

Key Insights and Ideas

  • The Label Distribution Gap: The biggest "aha" moment is the realization that standard HUDA methods fail in PU settings because they assume the source has a balanced mix of labels. If the source is 100% positive, a standard alignment algorithm will try to make the entire target look like a positive class, which is counter-productive for classification.
  • Asymmetric Transformation: The authors found it more effective to only transform the target features toward the source space, rather than trying to map both to a new "latent" space. This reduces the complexity of the optimization problem.
  • Common Features as a "Warm-up": Even a small number of shared features (like age or gender) can act as a vital anchor. By using these to create a "teacher" model ($C_0$), the complex adversarial network learns much faster and avoids trivial solutions (like predicting everyone is negative).

Key Results

PADA and its soft-labeled variant, $PADA_S$, consistently outperformed baselines across all datasets:

  • Accuracy Boost: On the Movielens-Netflix dataset, PADA improved accuracy by about 4-6% over standard PU-learning and HUDA combinations.
  • Robustness to Imbalance: In the credit card dataset, even when only 10% of the target data was actually positive, $PADA_S$ maintained high performance, whereas other methods saw significant drops in AUC (Area Under the Curve).
  • Feature Alignment: Qualitative analysis showed that PADA successfully brought the "likely positive" target samples closer to the source data in the feature space while keeping the "likely negative" target samples far away.

Limitations and Future Work

The authors acknowledge that their current model assumes that the "common features" themselves don't have a massive distribution divergence between domains. If the shared features (like "income") are measured differently or have vastly different scales between the bank and the jewelry store, the "soft-labeling" anchor might fail.

Future work aims to incorporate techniques to reduce the divergence specifically within these common features to make the "teacher" model even more reliable.

Summary updated

Abstract

Heterogeneous unsupervised domain adaptation (HUDA) is the most challenging domain adaptation setting where the feature space differs between source and target domains, and the target domain has only unlabeled data. Existing HUDA methods assume that both positive and negative examples are available in the source domain, which may not be satisfied in some real applications. This paper addresses a new challenging setting called positive and unlabeled heterogeneous domain adaptation (PU-HDA), a HUDA setting where the source domain only has positives. PU-HDA can also be viewed as an extension of PU learning where the positive and unlabeled examples are sampled from different domains. A naive combination of existing HUDA and PU learning methods is ineffective in PU-HDA due to the gap in label distribution between the source and target domains. To overcome this issue, we propose a novel method, positive-adversarial domain adaptation (PADA), which can predict likely positive examples from the unlabeled target data and simultaneously align the feature spaces to reduce the distribution divergence between the whole source data and the likely positive target data. PADA achieves this by a unified adversarial training framework for learning a classifier to predict positive examples and a feature transformer to transform the target feature space to that of the source. Specifically, they are both trained to fool a common discriminator that determines whether the likely positive examples are from the target or source domain. We experimentally show that PADA outperforms several baseline methods, such as the naive combination of HUDA and PU learning.
BibTeX
@inproceedings{mori2023heterogeneous,
  author = {Junki Mori and Ryo Furukawa and Isamu Teranishi and Jun Sakuma},
  title = {{Heterogeneous Domain Adaptation with Positive and Unlabeled Data}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386193},
}

Loading related content...

Figure