Abstract
Unsupervised Domain Adaptation (UDA) has attracted a lot of attention in the last ten years. The emergence of Domain Invariant Representations (IR) has improved drastically the transferability of representations from a labelled source domain to a new and unlabelled target domain. However, a potential pitfall of this approach, namely the presence of label shift, has been brought to light. Some works address this issue with a relaxed version of domain invariance obtained by weighting samples, a strategy often referred to as Importance Sampling. From our point of view, the theoretical aspects of how Importance Sampling and Invariant Representations interact in UDA have not been studied in depth. In the present work, we present a bound of the target risk which incorporates both weights and invariant representations. Our theoretical analysis highlights the role of inductive bias in aligning distributions across domains. We illustrate it on standard benchmarks by proposing a new learning procedure for UDA. We observed empirically that weak inductive bias makes adaptation more robust. The elaboration of stronger inductive bias is a promising direction for new UDA algorithms.
Introduction: The Challenge of Label Shift
Machine learning models often fail when deployed in the real world because the target domain data differs from the source domain used for training. This "distributional shift" is a primary hurdle in Unsupervised Domain Adaptation (UDA). Traditional UDA techniques focus on learning domain-invariant representations—features that appear identical across domains regardless of their origin.
However, this approach faces a significant limitation: label shift. When the frequency of classes changes between domains (e.g., a balanced training set vs. a skewed target set), forcing perfect invariance can actually degrade model performance. This paper addresses this gap by investigating the interaction between importance sampling (weighting) and invariant representations, providing a more robust framework for adaptation.
Theoretical Framework: Uncoupling Invariance and Transferability
The authors derive a new generalization bound for target risk that explicitly incorporates weights and representations. This bound uncouples the error into two primary components:
- Invariance Error (INV): Quantifies the alignment between weighted source features and target features.
- Transferability Error (TSF): Measures the consistency of the feature-to-label relationship across domains.
Crucially, the analysis reveals that because target labels are unavailable, researchers must rely on inductive bias—assumptions about model structure or predicted labels—to estimate and minimize the transferability error. The authors argue that seeking "perfect" domain invariance is counterproductive if label distributions differ; it is more effective to align weighted distributions.
The RUDA Algorithm
To operationalize these theoretical insights, the authors propose Robust UDA (RUDA). RUDA employs a bi-level optimization strategy:
- Weight Learning: A domain discriminator learns weights that minimize the Invariance Error, effectively "relaxing" the invariance constraint to focus on the most relevant source samples for the target task.
- Classifier Training: Simultaneously, the classifier is trained to minimize the Transferability Error using the model's own predictions as a "weak" inductive bias.
Experimental Evaluation
The effectiveness of RUDA was tested on standard benchmarks including Digits (MNIST/USPS) and Office-31. To test robustness, the authors created "stress-test" versions of these datasets with artificial label shifts, reducing the frequency of certain classes in the source domain.
Key Results
- Robustness to Shift: On label-shifted stress tests, the weighted version (RUDAw) outperformed unweighted baselines by significant margins, including a +16.0% accuracy gain on Digits and +1.7% on Office-31.
- Competitive Performance: On standard, unshifted datasets, RUDA remains competitive with state-of-the-art methods like CDAN, despite its simpler scalar product architecture.
- The Power of Weak Bias: The results demonstrate that even a weak inductive bias (using predicted labels) significantly enhances robustness compared to methods that ignore label shift entirely.
Conclusion
This work provides a unified theoretical perspective on weighting and representations in UDA. By demonstrating that "perfect" invariance can be a trap under label shift, the authors highlight the necessity of importance sampling and inductive bias. While the authors note that weighting is most beneficial under significant shift—and can occasionally lead to slight degradation when shift is minimal—the development of stronger inductive biases remains a promising path for future UDA research.
Summary updated