How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?

Zhong Li, Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, Guangquan Zhang

Abstract

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Theoretical bounds for UDA typically involve three terms: source risk, distribution discrepancy, and a combined risk term ($\lambda$). While most existing methods focus on minimizing the first two, they often assume $\lambda$ is a negligible constant. This paper challenges this assumption, demonstrating that combined risk fluctuates significantly during training and can impede performance. We propose E-MixNet, a novel framework that actively minimizes combined risk using an "enhanced mixup" (e-mixup) strategy and a robust proxy loss, achieving state-of-the-art results across multiple benchmarks.

Introduction

The core challenge of Unsupervised Domain Adaptation (UDA) is bridging the gap between a labeled source dataset and an unlabeled target dataset. Theoretical analysis suggests that the target error is bounded by the source error, the discrepancy between domain distributions, and the "ideal" joint error, known as the combined risk ($\lambda$). Historically, researchers have treated $\lambda$ as a fixed, small constant. However, in deep learning, the feature representation is learned rather than fixed. As the feature space evolves, the combined risk can increase, particularly if the model maps different classes from different domains to the same feature space regions.

Rethinking Combined Risk

The authors prove that combined risk is not a constant but a dynamic variable tied to the conditional distribution discrepancy. If a feature transformation collapses distinct class boundaries across domains, the "ideal" shared classifier will inevitably fail, causing $\lambda$ to spike.

Combined risk and accuracy trends over training epochs. (a) shows how combined risk fluctuates for different methods, while (b) illustrates the corresponding impact on accuracy.

As shown in Figure 3, methods that ignore combined risk (like standard pseudo-labeling or mixup) often see it rise as training progresses, which correlates with plateauing or declining accuracy. E-MixNet (e-mixup) maintains a lower and more stable combined risk throughout the training process.

The E-MixNet Framework

To address this, the authors introduce E-MixNet, designed to explicitly minimize the combined risk proxy.

Enhanced Mixup (e-mixup)

Minimizing combined risk requires labels for both domains. Since target labels are unavailable, E-MixNet utilizes pseudo-labels. To improve robustness against noisy pseudo-labels, the authors propose e-mixup. Unlike standard mixup, e-mixup specifically selects pairs of samples that share the same (pseudo) label but originate from different domains or distant feature regions. This encourages the model to learn a domain-invariant representation that respects class boundaries.

Proxy Loss and Architecture

The framework employs a dedicated classifier ($C^*$) to estimate the combined risk. Instead of standard cross-entropy, the authors use Mean Squared Error (MSE) for the combined risk proxy. MSE is bounded and provides smoother gradients, making it less sensitive to the incorrect pseudo-labels common in early training stages.

Architecture of the E-MixNet framework compared to general UDA models. The right panel details the proposed triple-branch structure for source risk, discrepancy, and combined risk proxy.

Experimental Results

E-MixNet was evaluated on three major UDA benchmarks: Office-31, Image-CLEF, and Office-Home.

  • Office-31: Achieved an average accuracy of 90.2%, significantly outperforming the MDD baseline, particularly on the challenging DSLR $\to$ Amazon task.
  • Image-CLEF: Set a new state-of-the-art with 91.0% average accuracy.
  • Office-Home: On this complex 65-class dataset, E-MixNet reached 70.6%, a notable improvement over the previous best of 68.1%.

The authors also demonstrated that their combined risk proxy is "plug-and-play." Integrating it into existing architectures like DANN or CDAN consistently yielded performance gains, validating the theoretical importance of $\lambda$.

Conclusion and Future Work

This research highlights a critical oversight in UDA: the non-constant nature of combined risk. By introducing E-MixNet and the e-mixup strategy, the authors provide a practical way to monitor and minimize this risk. While the method is sensitive to initial pseudo-label quality, it represents a significant step toward more theoretically grounded and high-performing domain adaptation. Future work may explore self-supervised pre-training to further stabilize the initial pseudo-labeling phase.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) aims to train a target classifier with labeled samples from the source domain and unlabeled samples from the target domain. Classical UDA learning bounds show that target risk is upper bounded by three terms: source risk, distribution discrepancy, and combined risk. Based on the assumption that the combined risk is a small fixed value, methods based on this bound train a target classifier by only minimizing estimators of the source risk and the distribution discrepancy. However, the combined risk may increase when minimizing both estimators, which makes the target risk uncontrollable. Hence the target classifier cannot achieve ideal performance if we fail to control the combined risk. To control the combined risk, the key challenge takes root in the unavailability of the labeled samples in the target domain. To address this key challenge, we propose a method named E-MixNet. E-MixNet employs enhanced mixup, a generic vicinal distribution, on the labeled source samples and pseudo-labeled target samples to calculate a proxy of the combined risk. Experiments show that the proxy can effectively curb the increase of the combined risk when minimizing the source risk and distribution discrepancy. Furthermore, we show that if the proxy of the combined risk is added into loss functions of four representative UDA methods, their performance is also improved.
BibTeX
@inproceedings{li2021how,
  author = {Zhong Li and Zhen Fang and Feng Liu and Jie Lu and Bo Yuan and Guangquan Zhang},
  title = {{How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i12.17322},
}
Figure