Generalized Domain Adaptation with Covariate and Label Shift CO-ALignment

Shuhan Tan, Xingchao Peng, Kate Saenko

Abstract

Unsupervised domain adaptation is a promising way to generalize deep models to novel domains. However, the current literature assumes that the label distribution is domain-invariant and only aligns the feature distributions or vice versa. In this work, we explore the more realistic task of Class-imbalanced Domain Adaptation: How to align feature distributions across domains while the label distributions of the two domains are also different? Taking a practical step towards this problem, we constructed the first benchmark with 22 cross-domain tasks from 6 real-image datasets. We conducted comprehensive experiments on 10 recent domain adaptation methods and find most of them are very fragile in the face of coexisting feature and label distribution shift. Towards a better solution, we further proposed a feature and label distribution CO-ALignment (COAL) model with a novel combination of existing ideas. COAL is empirically shown to outperform the most recent domain adaptation methods on our benchmarks. We believe the provided benchmarks, empirical analysis results, and the COAL baseline could stimulate and facilitate future research towards this important problem.

Introduction: Beyond Invariant Label Distributions

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Traditionally, UDA methods operate under the assumption that while the feature distribution $p(x|y)$ shifts (covariate shift), the label distribution $p(y)$ remains invariant across domains. However, in real-world scenarios, this assumption rarely holds. For instance, an autonomous vehicle trained in a city will encounter a vastly different frequency of pedestrians compared to one operating in a rural environment.

When both features and labels shift simultaneously—a problem termed Class-imbalanced Domain Adaptation (CDA)—standard UDA methods often suffer from negative transfer. By attempting to align the global marginal distributions, these methods inadvertently force dominant classes in the source domain to overlap with different dominant classes in the target domain, leading to misclassification.

Visualizing the challenge of Class-imbalanced Domain Adaptation and the COAL solution.

The COAL Framework

To address the dual challenges of feature and label shift, we propose COAL (CO-ALignment of feature and label distribution). COAL integrates three key components to ensure robust transfer:

  1. Prototype-based Conditional Alignment: Instead of marginal alignment, COAL performs conditional alignment. It utilizes a similarity-based classifier to establish "prototypes" for each class. By employing a minimax entropy approach, the model moves source prototypes toward target samples, ensuring that class-specific features are aligned correctly regardless of their frequency.
  2. Class-balanced Self-training: To estimate the target domain's label distribution, COAL uses pseudo-labels. Crucially, it employs a class-wise selection strategy, picking the "top $k$%" of confident samples per category. This prevents "easy" or frequent classes from overwhelming the adaptation process.
  3. Source Balanced Sampler: A mini-batch sampler ensures that every class is represented equally during training, preventing the model from inheriting the source domain's class biases.
The two-step COAL architecture: Step A for pseudo-labeling and Step B for feature/label co-alignment.

Experimental Benchmarking

We established a comprehensive benchmark for CDA consisting of 22 cross-domain tasks across several datasets:

  • Digits: MNIST, USPS, SVHN, and Synthetic Digits.
  • Office-Home: 65 categories across four domains (Real World, Clipart, Product, Art).
  • DomainNet: 40 common classes across four domains (Real, Clipart, Painting, Sketch).
  • RS-UT Protocol: A "Reversely-unbalanced Source and Unbalanced Target" protocol designed to simulate realistic, conflicting class imbalances.

Key Findings and Results

Our empirical odyssey revealed a significant gap in current research. On the Digits benchmark, 8 out of 10 state-of-the-art domain adaptation methods performed worse than a simple "Source Only" baseline. This underscores the fragility of existing methods when faced with label shift.

In contrast, COAL demonstrated superior performance and robustness:

  • Accuracy: COAL achieved an average accuracy of 84.33% on Digits tasks, significantly outperforming the next best method (DANN) at 75.98%.
  • Stability: While competitors like MCD saw performance plummet as label imbalance increased, COAL's accuracy remained stable across varying degrees of shift.
  • Feature Separation: t-SNE visualizations confirmed that COAL maintains distinct, well-separated clusters for each category in the target domain, whereas previous methods often mixed different classes during alignment.

Conclusion

Class-imbalanced Domain Adaptation represents a critical step toward deploying robust AI in the real world. By co-aligning feature and label distributions, COAL provides a strong baseline for this challenging task. Our findings suggest that future research must move beyond marginal alignment and explicitly account for the dynamic nature of class distributions across domains.

Summary updated

Abstract

Unsupervised domain adaptation is a promising way to generalize deep models to novel domains. However, the current literature assumes that the label distribution is domain-invariant and only aligns the feature distributions or vice versa. In this work, we explore the more realistic task of Class-imbalanced Domain Adaptation: How to align feature distributions across domains while the label distributions of the two domains are also different? Taking a practical step towards this problem, we constructed the first benchmark with 22 cross-domain tasks from 6real-image datasets. We conducted comprehensive experiments on 10 recent domain adaptation methods and find most of them are very fragile in the face of coexisting feature and label distribution shift. Towards a better solution, we further proposed a feature and label distribution CO-ALignment (COAL) model with a novel combination of existing ideas. COAL is empirically shown to outperform the most recent domain adaptation methods on our benchmarks. We believe the provided benchmarks, empirical analysis results, and the COAL baseline could stimulate and facilitate future research towards this important problem.
BibTeX
@inproceedings{tan2020generalized,
  author = {Shuhan Tan and Xingchao Peng and Kate Saenko},
  title = {{Generalized Domain Adaptation with Covariate and Label Shift CO-ALignment}},
  booktitle = {Workshops @ ECCV},
  year = {2020},
  url = {https://openreview.net/forum?id=0vzc4228o1},
}
Figure