Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, Jinwoo Shin

Abstract

While semi-supervised learning (SSL) has proven to be a promising way for leveraging unlabeled data when labeled data is scarce, the existing SSL algorithms typically assume that training class distributions are balanced. However, these SSL algorithms trained under imbalanced class distributions can severely suffer when generalizing to a balanced testing criterion, since they utilize biased pseudo-labels of unlabeled data toward majority classes. To alleviate this issue, we formulate a convex optimization problem to softly refine the pseudo-labels generated from the biased model, and develop a simple algorithm, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently. Under various class-imbalanced semi-supervised scenarios, we demonstrate the effectiveness of DARP and its compatibility with state-of-the-art SSL schemes.

Introduction: The Bias Feedback Loop

Semi-supervised learning (SSL) has revolutionized deep learning by enabling models to learn from vast amounts of unlabeled data. Modern techniques like MixMatch, ReMixMatch, and FixMatch rely heavily on pseudo-labeling, where the model's own predictions serve as training targets for unlabeled samples. However, a critical assumption underlies these methods: class balance.

In real-world scenarios, data often follows a long-tailed distribution. When SSL is applied to such imbalanced datasets, a destructive feedback loop emerges. The model, initially biased toward majority classes by the labeled data, generates pseudo-labels that are even more skewed. For instance, on a CIFAR-10 dataset with a 150:1 imbalance ratio, standard MixMatch can produce pseudo-labels with an effective imbalance exceeding 1000:1. This bias reinforcement leads to a significant performance drop, sometimes causing SSL models to perform worse than "vanilla" models trained only on the limited labeled set.

The impact of class imbalance on SSL: (a) training distribution, (b) resulting pseudo-label bias, and (c) accuracy gain degradation.

Methodology: The DARP Refinery

To address this, we propose Distribution Aligning Refinery of Pseudo-label (DARP). Rather than a new SSL algorithm, DARP acts as a model-agnostic post-processing "refinery" that corrects biased pseudo-labels before they are used for training.

Convex Optimization Framework

DARP formulates pseudo-label refinement as a convex optimization problem. The objective is to find a refined set of pseudo-labels that satisfy two primary constraints:

  1. Distribution Alignment: The aggregate distribution of the refined pseudo-labels must match the expected class distribution of the unlabeled data.
  2. Minimal Distortion: The refined labels should remain as close as possible to the model's original predictions, typically measured via KL-divergence. This ensures that the model's learned feature representations are preserved.

Efficient Solving and Noise Handling

The optimization is solved efficiently using a Dual Coordinate Ascent algorithm, an iterative process similar to the Sinkhorn algorithm used in optimal transport. To improve robustness, we introduce a noise removal hyperparameter ($\delta$) that masks out low-confidence, potentially noisy predictions, focusing the refinement on the most reliable data points. Furthermore, when the true distribution of unlabeled data is unknown, DARP can estimate it using a confusion matrix derived from a small validation set.

Experimental Evaluation

We evaluated DARP across several benchmarks, including CIFAR-10, CIFAR-100, STL-10, and the large-scale SUN-397 dataset. In all cases, DARP was "plugged in" to existing state-of-the-art SSL frameworks.

Performance comparison on CIFAR-10 (labeled imbalance ratio = 100) across varying unlabeled imbalance ratios. DARP consistently improves SOTA SSL methods.

The results demonstrate that DARP consistently and significantly improves performance. On CIFAR-10, adding DARP to MixMatch resulted in a relative reduction in balanced test error of up to 77.2%. Similar gains were observed for ReMixMatch (31.4%) and FixMatch (53.1%). Notably, as the data imbalance becomes more severe, the performance benefits of DARP become even more pronounced.

Conclusion

DARP provides a simple, efficient, and theoretically grounded solution to the problem of class imbalance in semi-supervised learning. By breaking the bias feedback loop through distribution alignment, it allows existing SSL frameworks to maintain their effectiveness in real-world, long-tailed data scenarios. Future work will explore DARP's performance in "open-set" scenarios where unlabeled data may contain classes not present in the labeled training set.

Summary updated

Abstract

While semi-supervised learning (SSL) has proven to be a promising way for leveraging unlabeled data when labeled data is scarce, the existing SSL algorithms typically assume that training class distributions are balanced. However, these SSL algorithms trained under imbalanced class distributions can severely suffer when generalizing to a balanced testing criterion, since they utilize biased pseudo-labels of unlabeled data toward majority classes. To alleviate this issue, we formulate a convex optimization problem to softly refine the pseudo-labels generated from the biased model, and develop a simple algorithm, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently. Under various class-imbalanced semi-supervised scenarios, we demonstrate the effectiveness of DARP and its compatibility with state-of-the-art SSL schemes.
BibTeX
@inproceedings{kim2020distribution,
  author = {Jaehyung Kim and Youngbum Hur and Sejun Park and Eunho Yang and Sung Ju Hwang and Jinwoo Shin},
  title = {{Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=echElM5ymjs},
}
Figure