Importance Weighting for Label Shift Adaptation

30 concepts27 papers1 proposal

Imagine training a machine learning model to diagnose a rare medical condition. In your training dataset, collected from a specialized research hospital, 50% of the patients have the condition. But when you deploy the model in a general clinic, only 0.1% of the patients have it.

Even if the physical symptoms of the condition look identical in both hospitals, your model will fail. It will drastically over-predict the condition because it assumes the baseline probability is 50%.

This scenario is a classic example of label shift (also known as target shift). Formally, label shift occurs when the marginal distribution of labels changes between the training (source) domain and the deployment (target) domain ($P_{\text{source}}(Y) \neq P_{\text{target}}(Y)$), while the conditional distribution of features given the label remains constant ($P_{\text{source}}(X|Y) = P_{\text{target}}(X|Y)$).

While traditional domain adaptation often focuses on covariate shift (where feature frequencies change but the decision boundary stays the same), label shift presents a unique challenge: we must correct for a mismatch in class frequencies without ever seeing the labels in the target domain.

Why Naive Feature Invariance Fails

A common strategy in domain adaptation is to learn "domain-invariant" representations—transforming the data into a latent space where the source and target feature distributions look identical. However, as You-Wei Luo et al. theoretically demonstrate, standard invariant representation learning is fundamentally insufficient when label distributions shift.

When class proportions differ, forcing the global feature distributions to match across domains creates a destructive phenomenon known as negative transfer. If the target domain is dominated by Class A, but the source domain has an equal mix of Class A and Class B, an adversarial discriminator will force the feature extractor to blur the boundaries between the classes just to make the overall distributions overlap.

As Ehsan Imani et al. point out, instead of twisting feature representations to match across domains, a more robust strategy is to leave the features intact and adjust the classifier itself. To do this safely, we must compute importance weights:

$$w(y) = \frac{P_{\text{target}}(y)}{P_{\text{source}}(y)}$$

If we can accurately estimate these weights, we can rebalance our source training data or adjust our model's output probabilities to match the target environment perfectly.

Estimating the Unseen: The Confusion Matrix Trick

The core hurdle of label shift adaptation is estimating $P_{\text{target}}(Y)$ when we only have access to unlabeled target features $X$. How do we count classes we cannot see?

A foundational breakthrough by Zachary C. Lipton et al. introduced Black Box Shift Estimation (BBSE). The intuition relies on a simple insight: if we know how often a model confuses certain classes on labeled validation data, we can use those error rates to back out the true class proportions from the model's target predictions.

Imagine a binary classifier. On validation data, we compute its confusion matrix $C$, where $C_{ij} = P(\hat{Y}=i \mid Y=j)$. When we deploy this model on the unlabeled target data, we observe an average predicted distribution $P_{\text{target}}(\hat{Y})$. Because of the law of total probability, the observed predictions are simply the true target label proportions filtered through the model's confusion matrix:

$$P_{\text{target}}(\hat{Y}) = C \cdot P_{\text{target}}(Y)$$

Because we can calculate $C$ from the source data and observe $P_{\text{target}}(\hat{Y})$ directly, we can solve this linear system to isolate the unknown true target distribution:

$$P_{\text{target}}(Y) = C^{-1} \cdot P_{\text{target}}(\hat{Y})$$

[ True Target Labels P(Y) ] ---> [ Confusion Matrix C ] ---> [ Observed Predictions P(Ŷ) ]
             ^                                                            |
             |____________________ ( Invert C ) __________________________|

This approach scales linearly with sample size, making it highly efficient for deep learning. Building on this foundation, Matthieu Kirchmeyer et al. and Chao Xu et al. have extended confusion-matrix estimation to more complex scenarios, such as generalized target shifts and cross-modal knowledge transfer, by formulating the inversion as a regularized quadratic program to ensure stability.

Alternative: Latent Space Matching

An alternative way to estimate target class proportions is to bypass the confusion matrix and match distributions directly in a shared latent space. As Kun Zhang et al. originally proposed, one can use kernel mean embeddings to find the importance weights that best align the source and target feature distributions.

Similarly, Yitong Li et al. demonstrate that target label proportions can be estimated by minimizing the distance between the marginal distributions of a model's latent features. By treating the target distribution as a mixture of source class conditional distributions, the model can iteratively solve for the mixture weights that make the source and target distributions overlap perfectly.

Correcting the Model

Once we have estimated the target label proportions, we must apply these importance weights to correct the model's behavior. This correction typically happens at two levels: loss reweighting and representation alignment.

1. Loss and Prediction Rectification

The most direct application is to modify the training loss or the final prediction probabilities. Jinhong Deng et al. apply this to cross-domain semantic segmentation, where object frequencies (like "sidewalk" vs. "car") vary wildly between synthetic training games and real-world streets. By scaling the training loss for each pixel by its corresponding class importance weight, the model avoids developing a bias toward over-represented source classes.

In unsupervised settings where models generate "pseudo-labels" for target data, naive models often fall into a feedback loop where easy-to-predict classes dominate. To break this cycle, Yunlong Zhang et al. use class distribution calibration to artificially magnify the probability scores of under-predicted, "hard" classes, ensuring balanced representation during training.

2. Weight-Aware Domain Alignment

If you do want to learn domain-invariant features, importance weights act as a crucial safety valve. Ruicheng Xian et al. and Didi Zhu et al. incorporate importance weights directly into adversarial domain alignment losses.

Instead of forcing the global target distribution to match the global source distribution, the weights re-scale the source domain mini-batches during training to match the estimated target proportions (Sehyun Hwang et al.). As Victor Bouvier et al. and Alain Rakotomamonjy et al. demonstrate, this rebalancing allows the feature extractor to achieve true semantic invariance without distorting or compressing the underlying feature representations.

Expanding the Horizon

The principles of label shift correction are expanding beyond standard discrete classification tasks:

  • Continuous Spaces: In tasks like gaze estimation, labels are continuous coordinates rather than discrete classes. Hao-Ran Yang et al. solved this by modeling continuous label spaces using truncated probability distributions, allowing importance reweighting to function smoothly for regression.
  • Multi-Task Learning: When training a model on multiple tasks simultaneously, class frequencies often differ from task to task. Fan Zhou et al. showed that multi-task performance is highly sensitive to this internal label shift, and that incorporating a class-frequency reweighting mechanism prevents dominant tasks from overshadowing minority classes.

By decoupling the distribution of labels from the underlying structural relationships of the data, importance weighting provides a mathematically rigorous bridge for models moving from controlled training environments into an unbalanced, shifting world.

Key Foundational Work

Re-Weighted Adversarial Adaptation Network for Unsupervised Domain Adaptation (2018) — contributed 2 concept elements to this direction.
Domain Adaptation under Target and Conditional Shift (2013) — contributed 1 concept elements to this direction.
Detecting and Correcting for Label Shift with Black Box Predictors (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_class_frequency_re_weighting_ae9d6027,
  title = {class-frequency re-weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-frequency-re-weighting/rdr_ae9d602748db4eecab5e3bb2c9b0163f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_class_frequency_re_weighting_ae9d6027_v1,
  title = {class-frequency re-weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-frequency-re-weighting/rdr_ae9d602748db4eecab5e3bb2c9b0163f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake