Unsupervised Domain Adaptation via Minimized Joint Error

Dexuan Zhang, Thomas Westfechtel, Tatsuya HaradaTMLR 2023 · OpenReview

Abstract

Unsupervised domain adaptation transfers knowledge from a fully labeled source domain to a different target domain, where no labeled data are available. Some researchers have proposed upper bounds for the target error when transferring knowledge. For example, Ben-David et al. (2010) established a theory based on minimizing the source error and distance between marginal distributions simultaneously. However, in most research, the joint error is ignored because of its intractability. In this research, we argue that joint errors are essential for domain adaptation problems, particularly when the domain gap is large. To address this problem, we propose a novel objective related to the upper bound of the joint error. Moreover, we adopt a source/pseudo-target label-induced hypothesis space that can reduce the search space to further tighten this bound. To measure the dissimilarity between hypotheses, we define a novel cross-margin discrepancy to alleviate instability during adversarial learning. In addition, we present extensive empirical evidence showing that the proposed method boosts the performance of image classification accuracy on standard domain adaptation benchmarks.

Introduction

In machine learning, the standard assumption is that training (source) and test (target) data share the same distribution. In real-world scenarios, this assumption often fails. Unsupervised Domain Adaptation (UDA) aims to bridge this gap by transferring knowledge from a labeled source domain to an unlabeled target domain.

Most existing UDA methods focus on marginal distribution alignment, attempting to make feature representations indistinguishable to a discriminator. However, as shown in Figure 1, aligning distributions without considering class labels (the "joint error") can lead to incorrect mappings, such as mapping "Class A" in the source to "Class B" in the target. While standard theories acknowledge the existence of joint error, it is often treated as a constant due to its intractability. This paper argues that for significant domain gaps, the joint error is variable and critical for model success.

Illustration of joint error and distribution matching. (a) Legend. (b) Joint error can increase even if marginal distributions match. (c) Target classifier location.

Methodology: Minimized Joint Error (MJE)

The proposed MJE framework rests on three pillars:

  1. Optimizable Upper Bound: The authors derive a new mathematical upper bound for target error that explicitly includes optimizable joint error terms.
  2. Target-driven Hypothesis Space (THS): To constrain the search space, the authors define a subspace of models that perform well on both labeled source data and pseudo-labeled target data. This prevents the model from converging to regions with high joint error.
  3. Cross-Margin Discrepancy (CMD): Adversarial training is often unstable. CMD is introduced as a novel discrepancy measure between hypotheses. Inspired by margin theory, CMD provides a smoother gradient profile compared to standard losses like Hinge or Logistic loss (see Figure 2). This "flatter" profile prevents unstable jumps during the minimax optimization process.
Loss curve comparison between Cross-Margin Discrepancy (CMD) and standard loss functions like Hinge and Logistic.

Experimental Results

The MJE method was evaluated on several benchmarks:

  • Digits (MNIST, SVHN, USPS): On the difficult MNIST → SVHN task, MJE achieved 50.3% accuracy, a massive improvement over the strong MCD baseline (11.2%).
  • VisDA-2017: MJE reached 81.6% average accuracy, significantly outperforming DANN (57.4%) and MCC (78.8%).
  • Office-Home & Office-31: MJE showed consistent improvements, though gains on Office-31 were more modest due to noise in the Amazon domain.

Conclusion

The success of MJE highlights that joint error is a dynamic and essential component of domain adaptation. By leveraging a target-driven hypothesis space and the stable Cross-Margin Discrepancy, the proposed method provides a more robust framework for UDA, particularly when dealing with large domain shifts and label imbalances.

Summary updated

Abstract

Unsupervised domain adaptation transfers knowledge from a fully labeled source domain to a different target domain, where no labeled data are available. Some researchers have proposed upper bounds for the target error when transferring knowledge. For example, Ben-David et al. (2010) established a theory based on minimizing the source error and distance between marginal distributions simultaneously. However, in most research, the joint error is ignored because of its intractability. In this research, we argue that joint errors are essential for domain adaptation problems, particularly when the domain gap is large. To address this problem, we propose a novel objective related to the upper bound of the joint error. Moreover, we adopt a source/pseudo-target label-induced hypothesis space that can reduce the search space to further tighten this bound. To measure the dissimilarity between hypotheses, we define a novel cross-margin discrepancy to alleviate instability during adversarial learning. In addition, we present extensive empirical evidence showing that the proposed method boosts the performance of image classification accuracy on standard domain adaptation benchmarks.
BibTeX
@article{zhang2023unsupervised,
  author = {Dexuan Zhang and Thomas Westfechtel and Tatsuya Harada},
  title = {{Unsupervised Domain Adaptation via Minimized Joint Error}},
  journal = {Accepted by TMLR},
  year = {2023},
  url = {https://openreview.net/forum?id=kiPsMct7vL},
}
Figure