One-step Label Shift Adaptation via Robust Weight Estimation

Ruidong Fan, Xiao Ouyang, Tingjin Luo, Lijun Zhang, Chenping HouIJCAI 2025 · DOI

Summary

Machine learning models often degrade when the distribution of labels changes between training and deployment, a phenomenon known as label shift. Traditional approaches address this via a disjointed two-step process: estimating importance weights and then retraining the classifier. This paper introduces OLSA (One-step Label Shift Adaptation), a bi-level optimization framework that learns the classifier and weights simultaneously. By incorporating a novel debiasing term and weight regularization, OLSA demonstrates superior robustness and accuracy in highly skewed environments compared to existing state-of-the-art methods.

1. Introduction: The Challenge of Label Shift

In real-world applications, data distributions are rarely stationary. A prominent challenge is label shift, where the conditional distribution of features given a label ($P(X|Y)$) remains constant, but the prior probability of the labels ($P(Y)$) changes between the source (training) and target (test) domains. For instance, a bird identification system might be trained on data where parrots are common but deployed in an environment dominated by egrets.

Visual representation of label shift in the data space versus the label distribution space.

Standard solutions typically rely on a two-step strategy: first, using a pre-trained classifier to estimate importance weights (the ratio of target to source label frequencies), and second, using these weights to calibrate the classifier. However, this approach ignores the interdependence between weight estimation and classifier performance. If the initial classifier is biased, the weight estimates become inaccurate, leading to a cycle of performance degradation.

2. The OLSA Framework

OLSA addresses these flaws by treating weight estimation and classifier training as a unified bi-level optimization problem, allowing the model to reach a "dynamic equilibrium."

2.1 Bi-level Optimization

The framework consists of two nested tasks:

  • Upper-level Task: The model trains the target classifier by minimizing an unbiased target risk. A key innovation here is a debiasing term that utilizes both labeled source data and unlabeled target data to ensure the classifier remains robust to distribution changes.
  • Lower-level Task: The model estimates importance weights based on the current classifier's predictions. To prevent instability early in training, a regularization term is added to keep the estimated distribution balanced unless significant evidence suggests otherwise.

2.2 Calibration and Optimization

Instead of standard cross-entropy, OLSA employs a tunable $\gamma$-loss during training. This loss "softens" output probabilities, providing more reliable inputs for weight estimation. The entire system is optimized using implicit gradients (IG), allowing the classifier updates to account for how those updates will change the weight estimation in subsequent steps.

3. Experimental Evaluation

The researchers evaluated OLSA against benchmarks including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. They simulated two primary types of shifts:

  1. Tweak-One Shift: Drastically altering the probability of a single class while keeping others proportional.
  2. Dirichlet Shift: A multi-class shift where proportions are drawn from a Dirichlet distribution (controlled by parameter $\alpha$).

4. Key Results and Insights

OLSA consistently outperformed state-of-the-art methods such as BBSE, MLLS, and ADM-OS across almost all scenarios:

  • High Skew Performance: On CIFAR-10 under a Dirichlet shift ($\alpha=0.1$), OLSA achieved a nearly 6% improvement in F-score over the next best method.
  • Robustness to Extreme Shifts: In "Tweak-One" scenarios with a shift factor of $\rho=0.9$, OLSA maintained a 4% F-score lead on Fashion-MNIST, where other methods struggled to adapt.
  • Weight Accuracy: Mean Squared Error (MSE) measurements confirmed that OLSA produces significantly more accurate weight estimates, particularly as the severity of the label shift increases.

5. Conclusion

OLSA represents a significant step forward in domain adaptation by unifying classifier training and weight estimation. While the bi-level optimization via implicit gradients increases computational complexity, the gains in robustness and accuracy—especially in identifying rare classes in highly skewed target domains—provide a compelling case for its adoption. Future work may focus on refining regularization techniques and reducing the computational overhead of the optimization process.

Summary updated

BibTeX
@inproceedings{fan2025one,
  author = {Ruidong Fan and Xiao Ouyang and Tingjin Luo and Lijun Zhang and Chenping Hou},
  title = {{One-step Label Shift Adaptation via Robust Weight Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/565},
}
Figure