LTF: A Label Transformation Framework for Correcting Label Shift

Jiaxian Guo, Mingming Gong, Tongliang Liu, Kun Zhang, Dacheng Tao

Introduction

The Label Transformation Framework (LTF) is an end-to-end neural network approach designed to correct "target shift"—a common scenario where the distribution of labels changes between training and testing, but the underlying relationship between features and labels remains stable. Unlike previous methods, LTF handles discrete, continuous, and multi-dimensional labels in a unified, scalable way by matching distributions in a low-dimensional feature space.

The Problem: Target Shift

In machine learning, we usually assume the training data (source) and test data (target) come from the same distribution. In the real world, this is rarely true. This paper focuses on target shift (also known as prior probability shift).

Imagine a flu-prediction model trained in a year with low infection rates. If you deploy it during a pandemic, the "marginal distribution" of the label (the probability of someone having the flu) has changed drastically, even if the symptoms (features) associated with the flu remain the same.

Existing solutions for target shift often struggle with three main issues:

  1. Scalability: Methods like Kernel Mean Matching are computationally expensive for large datasets.
  2. Label Types: Most methods only work for discrete categories (classification) and fail on continuous values (regression).
  3. High Dimensionality: When dealing with images, redundant information in the pixels can "drown out" the label-relevant signals, making it hard to estimate the shift accurately.

Proposed Method: The LTF Framework

The authors propose a three-part framework that uses the power of generative models to bridge the gap between domains:

  1. Label Transformation Network (LT): This network learns how to transform the source label distribution into a candidate target distribution. It essentially "guesses" how the labels have shifted.
  2. Label Influence Recovery Network (G): This is a conditional generator (similar to a GAN) that models the relationship between labels and features. It takes a label and some noise and generates a synthetic feature representation.
  3. Distribution Matching: The framework generates synthetic features using the "guessed" labels from the LT network. It then compares these synthetic features to the actual features observed in the unlabeled test set. By minimizing the difference between these two distributions (using adversarial training or Maximum Mean Discrepancy), the model learns the correct label shift.
The Label Transformation Framework (LTF) pipeline, illustrating the forward process and backpropagation through the Label Transformation and Influence Recovery networks.

The "Aha!" Moment: To handle high-dimensional data like images, the authors don't match raw pixels. Instead, they match distributions in a low-dimensional feature space ($h(X)$) that discards information irrelevant to the label $Y$. They prove theoretically that if this feature extractor is learned correctly, it satisfies the conditional independence needed for accurate shift estimation.

Datasets and Evaluation

The authors evaluated LTF across a variety of tasks to prove its versatility:

  • Discrete Labels (Classification): MNIST, Fashion-MNIST, and CIFAR-10. They simulated shifts like "Tweak-One" (boosting one class) and "Minority-Class" (suppressing certain classes).
  • Continuous Labels (Regression): A synthetic "Moon" dataset where the label is a continuous value along a curve.
  • Real-world Regression: The Sprites dataset, where the model predicts the 1D or 2D position of a character in an image. This tested the framework's ability to handle multi-dimensional continuous labels.

Key Results and Insights

By using neural networks to model the transformation, the authors moved away from rigid mathematical assumptions, allowing the same code to handle a 10-class classifier and a 2D coordinate regressor. Instead of trying to explicitly calculate the density ratio of labels (which is hard for continuous data), they implicitly model the process of generating features from labels.

  • Superior Accuracy: On CIFAR-10, LTF consistently outperformed state-of-the-art methods like BBSE and RLLS in estimating label weights, especially under extreme shifts like the Random Dirichlet shift.
Performance comparison on CIFAR-10 under Random Dirichlet Shift, showing LTF's superior accuracy and lower mean square error compared to baselines like BBSE and RLLS.
  • Continuous Success: In the Sprites 1D/2D position tasks, LTF was the first method shown to be compatible with large-scale multi-dimensional target shift, significantly reducing Mean Squared Error (MSE) compared to unadapted baselines.
  • Efficiency: The authors introduced a "quick adaptation" method where they only fine-tune the output layer of a pre-trained model using generated samples, achieving high performance without a full re-train.

Limitations and Future Work

While powerful, the framework has a few limitations:

  • GAN Training Hurdles: Like all adversarial frameworks, LTF relies on stable GAN training. If the conditional generator ($G$) fails to capture the feature-label relationship accurately, the shift estimation will be off.
  • Conditional Shift: The current framework assumes $P(X|Y)$ is invariant. In some real-world cases, both the labels and the way they manifest in features might change (conditional shift). The authors suggest extending LTF to handle these "joint" shifts in future work.
  • Hyperparameter Sensitivity: For discrete labels, the "Gumbel-softmax" trick requires tuning a temperature parameter, which can be tricky in unsupervised settings.

Summary updated

Abstract

Distribution shift is a major obstacle to the deployment of current deep learning models on real-world problems. Let 𝑌 be the class label and 𝑋 the features. We focus on one type of distribution shift, \emph{ label shift}, where the label marginal distribution 𝑃𝑌 changes but the conditional distribution 𝑃𝑋|𝑌 does not. Most existing methods estimate the density ratio between the source- and target-domain label distributions by density matching. However, these methods are either computationally infeasible for large-scale data or restricted to shift correction for discrete labels. In this paper, we propose an end-to-end Label Transformation Framework (LTF) for correcting label shift, which implicitly models the shift of 𝑃𝑌 and the conditional distribution 𝑃𝑋|𝑌 using neural networks. Thanks to the flexibility of deep networks, our framework can handle continuous, discrete, and even multi-dimensional labels in a unified way and is scalable to large data. Moreover, for high dimensional 𝑋, such as images, we find that the redundant information in 𝑋 severely degrades the estimation accuracy. To remedy this issue, we propose to match the distribution implied by our generative model and the target-domain distribution in a low-dimensional feature space that discards information irrelevant to 𝑌. Both theoretical and empirical studies demonstrate the superiority of our method over previous approaches.
BibTeX
@inproceedings{guo2020ltf,
  author = {Jiaxian Guo and Mingming Gong and Tongliang Liu and Kun Zhang and Dacheng Tao},
  title = {{LTF: A Label Transformation Framework for Correcting Label Shift}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure