When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights

You-Wei Luo, Chuan-Xian Ren

TL;DR

This paper demonstrates that standard invariant representation learning—the bread and butter of domain adaptation—is fundamentally insufficient when label distributions shift between environments. The authors provide rigorous proofs that explicit label shift correction is both theoretically necessary and sufficient for generalization, introducing a kernel-based algorithm (KECA) that significantly outperforms existing methods on standard benchmarks.

What Problem Does This Paper Address?

In classical machine learning, we assume training (source) and test (target) data come from the same distribution. In reality, "dataset shift" is common: a model trained on clean studio photos might fail on grainy smartphone images.

Most research tackles this using Invariant Representation Learning (IRL), which tries to find a feature space where the source and target look identical. However, most IRL methods assume the proportion of classes (e.g., the ratio of "cats" to "dogs") remains constant. When these proportions change—a phenomenon known as Label Shift—standard alignment techniques can actually "misalign" the data, forcing the model to make incorrect trade-offs. This paper explores the "Generalized Label Shift" (GLS) setting, where both the features and the label proportions change, and asks: Is feature alignment enough? (Spoiler: It isn't.)

Methods

The authors move beyond the standard model pair—a transformation ($g$) and a classifier ($h$)—to a triplet model $(g, h, w)$, where $w$ represents importance weights for labels.

Their proposed solution is the Kernel Embedding-based Correction Algorithm (KECA), which operates in two main stages:

  1. Weight Estimation ($w$): It uses Black Box Shift Estimation (BBSE) to calculate how much the label proportions have shifted, even though target labels are unknown during training.
  2. GLS Correction: It aligns distributions using Conditional Maximum Mean Discrepancy (CMMD) within a Reproducing Kernel Hilbert Space (RKHS). This ensures that "cats" in the source domain are aligned specifically with "cats" in the target domain, rather than just mashing all features together.
  3. Warm-up Strategy: To handle the "chicken-and-egg" problem of needing good labels to estimate shifts, they implement a warm-up phase on source data to ensure the initial pseudo-labels for the target domain are reliable.

Datasets

The authors evaluated their approach on three gold-standard computer vision benchmarks for domain adaptation:

  • Office-31: Images of office objects from three domains (Amazon, Webcam, DSLR).
  • Office-Home: A more complex dataset with 65 classes across four domains (Artistic, Clipart, Product, Real-World).
  • VisDA-2017: A large-scale challenge focusing on the shift from Synthetic (CAD models) to Real-world images.

To simulate real-world label shift, they subsampled specific classes in the source or target domains to create artificial imbalances.

Key Insights and Ideas

  • The Impossibility Result: One of the most striking findings (Proposition 1) is that if label shift exists, it is mathematically impossible to achieve both marginal invariance (making the whole feature clouds overlap) and conditional invariance (making class-specific clusters overlap) simultaneously. You must choose, and choosing marginal invariance leads to error.
  • IRL is Error-Prone: The paper proves that without explicit label correction, an invariant model will always have a non-zero "error floor" on the target domain, regardless of how much data you have.
  • The Bayesian Connection: They show that GLS correction isn't just a trick; it's theoretically sound from a Bayesian perspective, proving that a GLS-corrected learner is "sufficiently close" to the optimal target model.
  • Necessity of Weights: The research highlights that label weights ($w$) are not just an "extra feature" but a requirement to minimize the generalization bound when the environment changes.

Key Results

KECA consistently outperformed both standard invariant learning (like DANN and JAN) and existing label-shift correction methods.

  • Accuracy Gains: On the Office-Home dataset, KECA achieved an average accuracy of 65.9%, a massive jump over standard conditional alignment (CDAN) at 56.3%.
  • Large-Scale Performance: On the difficult VisDA-2017 task, KECA reached 72.4% accuracy, significantly higher than the 62.9% achieved by CDAN.
  • Qualitative Alignment: t-SNE visualizations showed that KECA produces much tighter, better-separated class clusters in the target domain compared to models that ignore label reweighting, which often suffered from "cluster misalignment."

Limitations and Future Work

The authors acknowledge that their method relies on pseudo-labels for the target domain to estimate conditional distributions. If the initial model is very poor, these pseudo-labels can be noisy, potentially leading to a "garbage in, garbage out" scenario (though their warm-up strategy mitigates this).

For future work, they suggest extending these GLS insights into Domain Generalization (where the target domain is completely unseen during training) and Source-Free Domain Adaptation (where the source data cannot be accessed due to privacy reasons).

Summary updated

Abstract

As a crucial step toward real-world learning scenarios with changing environments, dataset shift theory and invariant representation learning algorithm have been extensively studied to relax the identical distribution assumption in classical learning setting. Among the different assumptions on the essential of shifting distributions, generalized label shift (GLS) is the latest developed one which shows great potential to deal with the complex factors within the shift. In this paper, we aim to explore the limitations of current dataset shift theory and algorithm, and further provide new insights by presenting a comprehensive understanding of GLS. From theoretical aspect, two informative generalization bounds are derived, and the GLS learner are proved to be sufficiently close to optimal target model from the Bayesian perspective. The main results show the insufficiency of invariant representation learning, and prove the sufficiency and necessity of GLS correction for generalization, which provide theoretical supports and innovations for exploring generalizable model under dataset shift. From methodological aspect, we provide a unified view of existing shift correction frameworks, and propose a kernel embedding-based correction algorithm (KECA) to minimize the generalization error and achieve successful knowledge transfer. Both theoretical results and extensive experiment evaluations demonstrate the sufficiency and necessity of GLS correction for addressing dataset shift and the superiority of proposed algorithm.
BibTeX
@article{luo2024when,
  author = {You-Wei Luo and Chuan-Xian Ren},
  title = {{When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=q9RQ9KBLJJ},
}

Loading related content...

Figure