How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?

Li Zhong, Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, Guangquan Zhang

TL;DR

Most Unsupervised Domain Adaptation (UDA) methods focus on aligning distributions and minimizing source error, assuming the "combined risk" (the inherent difficulty of adapting between two specific domains) is a negligible constant. This paper proves that combined risk is actually dynamic and often increases during training; by introducing E-MixNet and a novel enhanced mixup (e-mixup) strategy, the authors provide a way to estimate and minimize this risk, leading to state-of-the-art performance across major benchmarks.

What Problem Does This Paper Address?

In Unsupervised Domain Adaptation (UDA), we have labeled data from a "source" domain (e.g., studio photos of chairs) and unlabeled data from a "target" domain (e.g., blurry webcam photos of chairs). The goal is to train a classifier that works on the target domain.

Theoretically, the error on the target domain is bounded by three terms:

  1. Source Risk: How well the model performs on the source data.
  2. Distribution Discrepancy: How different the source and target features look to the model.
  3. Combined Risk: The minimum possible error a single classifier could achieve on both domains simultaneously.

For years, researchers focused almost exclusively on the first two terms, assuming the combined risk was a small, fixed value. The authors of this paper show this is a mistake. They demonstrate that as you force a model to align two different distributions, the combined risk can actually spike, causing the model to fail on the target data even if it looks "aligned." The challenge is that we can't measure this risk directly because we don't have labels for the target domain.

Methods

The authors propose a new framework called E-MixNet to tackle this "unobservable" risk.

  • Theoretical Rethinking: They prove that when you use a feature transformer (like a neural network), the combined risk is no longer a constant—it becomes a function of that transformation.
  • e-mixup (Enhanced Mixup): To estimate the combined risk without target labels, they use a "proxy." While standard mixup creates new data points by interpolating between random pairs, e-mixup specifically pairs unlabeled target samples with "pure" labeled source samples. It also prioritizes pairing samples that are "distant" in the feature space.
  • The Proxy Loss: By training a dedicated "proxy" classifier ($C^*$) on these e-mixup samples using Mean Squared Error (MSE), the model can estimate how much the combined risk is growing.
  • Minimax Game: The training becomes a multi-objective optimization: minimize source error, minimize distribution discrepancy (using a discriminator), and minimize this new combined risk proxy.

Datasets

The researchers evaluated E-MixNet on three standard UDA benchmarks:

  • Office-31: 4,110 images across 31 categories from three domains (Amazon, DSLR, Webcam).
  • Office-Home: A more challenging set with 15,500 images across 65 categories (Artistic, Clipart, Product, Real-World).
  • Image-CLEF: A dataset containing 12 common classes shared across three domains (Caltech, ImageNet, Pascal).

Key Insights and Ideas

  • The "Aha" Moment: The authors visualized that as training progresses, the combined risk often goes up while accuracy goes down (the "green line" problem). This confirms that ignoring the third term of the learning bound is mathematically risky.
  • Source Anchoring: Why use source samples in the mixup? Because target pseudo-labels are often noisy. By mixing target samples with ground-truth source samples, the "e-mixup" strategy provides a stable anchor, preventing the model from drifting too far into incorrect predictions.
  • MSE over Cross-Entropy: They found that Mean Squared Error is a better loss function for the combined risk proxy because it is bounded and less sensitive to the "confidently wrong" predictions that often plague UDA pseudo-labeling.

Key Results

E-MixNet achieved impressive results, often outperforming competitive baselines like MDD and SymNets:

  • Office-31: Achieved an average accuracy of 90.2%, notably surpassing MDD (88.9%). On the difficult "DSLR to Amazon" task, it beat the baseline by 4.3%.
  • Office-Home: Reached 70.6% average accuracy, showing significant gains in difficult cross-domain tasks like "Artistic to Clipart."
  • Universal Improvement: Perhaps most impressively, the authors added their "combined risk proxy" to four other famous UDA methods (DAN, DANN, CDAN, SymNets). In every single case, the performance of those older methods improved—DANN, for instance, saw a 5.5% boost.

Limitations and Future Work

While the paper provides a robust solution, a few areas remain for exploration:

  • Hyperparameter Sensitivity: The method introduces parameters like $\alpha$ (for mixup) and $\gamma$ (for loss weighting), which may require careful tuning for new datasets.
  • Computational Cost: E-MixNet requires an additional classifier and a more complex training loop compared to simpler alignment methods.
  • Future Directions: The authors suggest that this approach to controlling combined risk could be extended to other adaptation scenarios, such as Partial Domain Adaptation (where the source and target don't share all the same classes) or Multi-source Adaptation.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) aims to train a target classifier with labeled samples from the source domain and unlabeled samples from the target domain. Classical UDA learning bounds show that target risk is upper bounded by three terms: source risk, distribution discrepancy, and combined risk. Based on the assumption that the combined risk is a small fixed value, methods based on this bound train a target classifier by only minimizing estimators of the source risk and the distribution discrepancy. However, the combined risk may increase when minimizing both estimators, which makes the target risk uncontrollable. Hence the target classifier cannot achieve ideal performance if we fail to control the combined risk. To control the combined risk, the key challenge takes root in the unavailability of the labeled samples in the target domain. To address this key challenge, we propose a method named E-MixNet. E-MixNet employs enhanced mixup, a generic vicinal distribution, on the labeled source samples and pseudo-labeled target samples to calculate a proxy of the combined risk. Experiments show that the proxy can effectively curb the increase of the combined risk when minimizing the source risk and distribution discrepancy. Furthermore, we show that if the proxy of the combined risk is added into loss functions of four representative UDA methods, their performance is also improved.
BibTeX
@inproceedings{zhong2021how,
  author = {Li Zhong and Zhen Fang and Feng Liu and Jie Lu and Bo Yuan and Guangquan Zhang},
  title = {{How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i12.17322},
}
Figure