Robust Domain Adaptation: Representations, Weights and Inductive Bias

Victor Bouvier, Philippe Very, Clément Chastagnol, Myriam Tami, Céline HudelotECML/PKDD 2020 · OpenReview

Abstract

Unsupervised Domain Adaptation (UDA) has attracted a lot of attention in the last ten years. The emergence of Domain Invariant Representations (IR) has improved drastically the transferability of representations from a labelled source domain to a new and unlabelled target domain. However, a potential pitfall of this approach, namely the presence of label shift, has been brought to light. Some works address this issue with a relaxed version of domain invariance obtained by weighting samples, a strategy often referred to as Importance Sampling. From our point of view, the theoretical aspects of how Importance Sampling and Invariant Representations interact in UDA have not been studied in depth. In the present work, we present a bound of the target risk which incorporates both weights and invariant representations. Our theoretical analysis highlights the role of inductive bias in aligning distributions across domains. We illustrate it on standard benchmarks by proposing a new learning procedure for UDA. We observed empirically that weak inductive bias makes adaptation more robust. The elaboration of stronger inductive bias is a promising direction for new UDA algorithms.

Introduction: The Challenge of Label Shift

Machine learning models often fail when deployed in the real world because the target domain data differs from the source domain used for training. This "distributional shift" is a primary hurdle in Unsupervised Domain Adaptation (UDA). Traditional UDA techniques focus on learning domain-invariant representations—features that appear identical across domains regardless of their origin.

However, this approach faces a significant limitation: label shift. When the frequency of classes changes between domains (e.g., a balanced training set vs. a skewed target set), forcing perfect invariance can actually degrade model performance. This paper addresses this gap by investigating the interaction between importance sampling (weighting) and invariant representations, providing a more robust framework for adaptation.

Theoretical Framework: Uncoupling Invariance and Transferability

The authors derive a new generalization bound for target risk that explicitly incorporates weights and representations. This bound uncouples the error into two primary components:

  1. Invariance Error (INV): Quantifies the alignment between weighted source features and target features.
  2. Transferability Error (TSF): Measures the consistency of the feature-to-label relationship across domains.

Crucially, the analysis reveals that because target labels are unavailable, researchers must rely on inductive bias—assumptions about model structure or predicted labels—to estimate and minimize the transferability error. The authors argue that seeking "perfect" domain invariance is counterproductive if label distributions differ; it is more effective to align weighted distributions.

The RUDA Algorithm

To operationalize these theoretical insights, the authors propose Robust UDA (RUDA). RUDA employs a bi-level optimization strategy:

  • Weight Learning: A domain discriminator learns weights that minimize the Invariance Error, effectively "relaxing" the invariance constraint to focus on the most relevant source samples for the target task.
  • Classifier Training: Simultaneously, the classifier is trained to minimize the Transferability Error using the model's own predictions as a "weak" inductive bias.

Experimental Evaluation

The effectiveness of RUDA was tested on standard benchmarks including Digits (MNIST/USPS) and Office-31. To test robustness, the authors created "stress-test" versions of these datasets with artificial label shifts, reducing the frequency of certain classes in the source domain.

t-SNE visualizations comparing feature alignment between RUDA (left) and standard invariant methods (right).

Key Results

  • Robustness to Shift: On label-shifted stress tests, the weighted version (RUDAw) outperformed unweighted baselines by significant margins, including a +16.0% accuracy gain on Digits and +1.7% on Office-31.
  • Competitive Performance: On standard, unshifted datasets, RUDA remains competitive with state-of-the-art methods like CDAN, despite its simpler scalar product architecture.
  • The Power of Weak Bias: The results demonstrate that even a weak inductive bias (using predicted labels) significantly enhances robustness compared to methods that ignore label shift entirely.

Conclusion

This work provides a unified theoretical perspective on weighting and representations in UDA. By demonstrating that "perfect" invariance can be a trap under label shift, the authors highlight the necessity of importance sampling and inductive bias. While the authors note that weighting is most beneficial under significant shift—and can occasionally lead to slight degradation when shift is minimal—the development of stronger inductive biases remains a promising path for future UDA research.

Summary updated

Abstract

Unsupervised Domain Adaptation (UDA) has attracted a lot of attention in the last ten years. The emergence of Domain Invariant Representations (IR) has improved drastically the transferability of representations from a labelled source domain to a new and unlabelled target domain. However, a potential pitfall of this approach, namely the presence of \textit{label shift}, has been brought to light. Some works address this issue with a relaxed version of domain invariance obtained by weighting samples, a strategy often referred to as Importance Sampling. From our point of view, the theoretical aspects of how Importance Sampling and Invariant Representations interact in UDA have not been studied in depth. In the present work, we present a bound of the target risk which incorporates both weights and invariant representations. Our theoretical analysis highlights the role of inductive bias in aligning distributions across domains. We illustrate it on standard benchmarks by proposing a new learning procedure for UDA. We observed empirically that weak inductive bias makes adaptation more robust. The elaboration of stronger inductive bias is a promising direction for new UDA algorithms.
BibTeX
@inproceedings{bouvier2020robust,
  author = {Victor Bouvier and Philippe Very and Clément Chastagnol and Myriam Tami and Céline Hudelot},
  title = {{Robust Domain Adaptation: Representations, Weights and Inductive Bias}},
  booktitle = {ECML/PKDD},
  year = {2020},
  url = {https://openreview.net/forum?id=JbdjlAiCGUM},
}
Figure