Unifying Distribution Alignment as a Loss for Imbalanced Semi-supervised Learning

Justin Lazarow, Kihyuk Sohn, Chen-Yu Lee, Chun-Liang Li, Zizhao Zhang, Tomas Pfister

Abstract

While remarkable progress has been made in imbalanced supervised learning, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only are few labeled data provided, but the underlying data distribution can be severely imbalanced. Recent work requires both complicated sampling strategies of pseudo-labeled unlabeled data and distribution alignment of the pseudo-label distribution to accommodate this imbalance. We present a novel approach that relies only on a form of a distribution alignment but no sampling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses. This alignment compensates for both imbalance in the data and the eventual distributional shift present during evaluation. Altogether, this provides a unified strategy that offers both significantly reduced training requirements and improved performance across both low and richly labeled regimes and over varying degrees of imbalance. In experiments, we validate the efficacy of our method on SSL variants of CIFAR10-LT, CIFAR100-LT, and ImageNet-127. On ImageNet-127, our method shows 1.6% accuracy improvement over CReST with an 80% training time reduction and is competitive with other SOTA methods.

Introduction

Semi-supervised learning (SSL) is a powerful paradigm for learning from limited labeled data and large unlabeled pools. However, standard SSL algorithms typically assume balanced class distributions. In real-world scenarios, data is often "long-tailed," where a few "head" classes have many examples while many "tail" classes have very few. Standard SSL methods fail here because models become heavily biased toward head classes, producing poor-quality pseudo-labels for the tail classes.

Previous solutions, such as CReST, attempted to mitigate this using "generational" approaches that involve training a model, sampling tail-class unlabeled data, and retraining from scratch. These multi-stage methods are computationally expensive and slow to converge. We propose UDAL (Unifying Distribution Alignment as a Loss), which simplifies imbalanced SSL by integrating distribution alignment directly into the training loss.

Methodology

UDAL treats distribution alignment not as a post-processing step or a sampling strategy, but as a core component of the loss function. The key technical insight is that Distribution Alignment (common in SSL) and Logit Adjustment (common in supervised learning) are mathematically related.

Comparison of UDAL's unified approach versus previous multi-stage SSL methods.

UDAL applies a log-offset to the model's logits during training to compensate for class imbalance. This adjustment is applied to both the supervised (labeled) and unsupervised (unlabeled) loss components. To ensure stable feature learning, UDAL employs a Progressive Alignment schedule. Early in training, the model focuses on the natural, imbalanced distribution to learn strong representations. As training progresses, the loss adjustment becomes more aggressive, forcing the classifier to produce more balanced predictions. Because it is a modification to the loss function, it requires no data resampling and can be added to existing SSL frameworks like FixMatch with minimal code changes.

Experimental Results

We evaluated UDAL on several standard long-tailed benchmarks, including CIFAR10-LT, CIFAR100-LT, and ImageNet-127.

Efficiency and Performance

A primary advantage of UDAL is its massive speedup. On ImageNet-127, UDAL reached convergence in 1,000 epochs—6x faster than the 6,000 epochs required by CReST+. On CIFAR datasets, it achieved a 5x reduction in training time.

Accuracy vs. training epochs on ImageNet-127, showing UDAL's superior convergence speed and final accuracy.

Beyond efficiency, UDAL delivers superior accuracy. On ImageNet-127, UDAL outperformed CReST+ by 1.6% in absolute accuracy. On CIFAR10-LT (with 10% labels and an imbalance ratio of 100), UDAL achieved 80.2% accuracy compared to FixMatch's 66.3%. Notably, on ImageNet-127, UDAL using only 10% labels came within 0.7% of the accuracy of a fully supervised model using 100% of the labels.

Conclusion

UDAL provides a unified strategy for imbalanced SSL that significantly reduces training requirements while improving performance across varying degrees of imbalance. By bridging the gap between supervised logit adjustment and semi-supervised distribution alignment, it offers a robust and efficient solution for real-world, long-tailed data. Future work includes exploring combinations with other imbalanced learning techniques and improving distribution estimation for cases where class priors are unknown.

Summary updated

Abstract

While remarkable progress has been made in imbalanced supervised learning, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only are few labeled data provided, but the underlying data distribution can be severely imbalanced. Recent work requires both complicated sampling strategies of pseudo-labeled unlabeled data and distribution alignment of the pseudo-label distribution to accommodate this imbalance. We present a novel approach that relies only on a form of a distribution alignment but no sampling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses. This alignment compensates for both imbalance in the data and the eventual distributional shift present during evaluation. Altogether, this provides a unified strategy that offers both significantly reduced training requirements and improved performance across both low and richly labeled regimes and over varying degrees of imbalance. In experiments, we validate the efficacy of our method on SSL variants of CIFAR10-LT, CIFAR100-LT, and ImageNet-127. On ImageNet-127, our method shows 1.6% accuracy improvement over CReST with an 80% training time reduction and is competitive with other SOTA methods. Code is available at https://github.com/google-research/crest
BibTeX
@inproceedings{lazarow2023unifying,
  author = {Justin Lazarow and Kihyuk Sohn and Chen-Yu Lee and Chun-Liang Li and Zizhao Zhang and Tomas Pfister},
  title = {{Unifying Distribution Alignment as a Loss for Imbalanced Semi-supervised Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=I1LlyIsvSbH},
}
Figure