An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning

Hao Chen

TL;DR

This paper introduces SimiS, a remarkably simple yet highly effective baseline for imbalanced semi-supervised learning (SSL). By strategically augmenting labeled data with pseudo-labels derived from unlabeled data, SimiS significantly reduces class imbalance, leading to state-of-the-art performance across various benchmarks and demonstrating impressive robustness to diverse data distributions.

What Problem Does This Paper Address?

Semi-supervised learning (SSL) aims to leverage abundant unlabeled data to improve model performance, typically assuming a balanced distribution of classes. However, real-world datasets often exhibit a long-tailed or imbalanced distribution, where a few classes have many samples (head classes) and many classes have very few (tail classes). This imbalance poses a significant challenge for standard SSL methods, leading to confirmation bias where models over-rely on majority classes, producing biased pseudo-labels and ultimately hurting generalization performance. Existing methods for imbalanced SSL struggle to effectively mitigate this imbalance, especially under severe imbalance ratios.

Methods

The core of SimiS lies in its simplicity: it augments the labeled dataset with pseudo-labels generated from unlabeled data. Specifically, after each training epoch, SimiS generates pseudo-labels for the unlabeled data and then supplements the infrequent (tail) classes in the labeled set with the most confident pseudo-labels. The number of pseudo-labels added for each tail class is determined by the difference in class distribution between the most frequent class and that specific tail class. This approach focuses on boosting the representation of tail classes without excessively adding to the head classes, thereby effectively reducing the class imbalance in the labeled set. The method is designed to be easily integrated with existing SSL algorithms.

Datasets

The authors evaluated SimiS on several imbalanced datasets:

  • CIFAR10-LT, CIFAR100-LT, STL10-LT: These are standard datasets with artificially created long-tailed distributions.
  • FOOD101-LT: Another dataset with an artificially created long-tailed distribution.
  • ImageNet127: A naturally long-tailed dataset derived from ImageNet.

Key Insights and Ideas

  • Simplicity is Key: The paper strongly advocates for the effectiveness of a simple approach. By focusing on reducing the imbalance in the labeled data, SimiS achieves remarkable results without complex mechanisms.
  • Reducing Labeled Data Imbalance is Sufficient: The experiments suggest that reducing the imbalance ratio of the labeled set is a crucial and often sufficient step for achieving strong performance in imbalanced SSL, even if the unlabeled data remains imbalanced.
  • Robustness to Distribution Shifts: SimiS demonstrates impressive robustness to various combinations of training and testing data distributions, a significant advantage over many existing methods that degrade under distribution shifts.
  • Flexibility and Compatibility: The straightforward nature of SimiS allows it to be easily combined with other SSL algorithms and re-balancing techniques, often leading to further performance improvements.

Key Results

  • State-of-the-Art Performance: SimiS consistently outperforms existing imbalanced SSL methods by a significant margin across various datasets and imbalance ratios. For instance, it achieves improvements of 12.8%, 13.6%, and 14.6% over previous SOTA on CIFAR100-LT, FOOD101-LT, and ImageNet127, respectively.
  • Faster Convergence and Better Pseudo-Label Accuracy: SimiS leads to faster convergence and improved accuracy of pseudo-labels, particularly for minority classes, which helps alleviate confirmation bias.
  • Robustness Demonstrated: Figures and tables show SimiS maintaining high accuracy across a wide range of imbalance ratios (γ) for both labeled (γl) and unlabeled (γu) data, unlike other methods that degrade significantly.
  • Synergy with Other Methods: When combined with techniques like Logits Adjustment (LA), SimiS further pushes the performance boundaries.

Limitations and Future Work

  • Lack of Formal Theoretical Analysis: The paper acknowledges that while empirical results are strong, a formal theoretical guarantee for SimiS's effectiveness is still needed.
  • Future Work: The authors hope their work will inspire further research, particularly on the theoretical underpinnings of SimiS, and encourage pushing the boundaries of imbalanced SSL for real-world deployment.

Summary updated

Abstract

Semi-supervised learning (SSL) has shown great promise in leveraging unlabeled data to improve model performance. While standard SSL assumes uniform data distribution, we consider a more realistic and challenging setting called imbalanced SSL, where imbalanced class distributions occur in both labeled and unlabeled data. Although there are existing endeavors to tackle this challenge, their performance degenerates when facing severe imbalance since they can not reduce the class imbalance sufficiently and effectively. In this paper, we study a simple yet overlooked baseline -- SimiS -- which tackles data imbalance by simply supplementing labeled data with pseudo-labels, according to the difference in class distribution from the most frequent class. Such a simple baseline turns out to be highly effective in reducing class imbalance. It outperforms existing methods by a significant margin, e.g., 12.8%, 13.6%, and 16.7% over previous SOTA on CIFAR100-LT, FOOD101-LT, and ImageNet127 respectively. The reduced imbalance results in faster convergence and better pseudo-label accuracy of SimiS. The simplicity of our method also makes it possible to be combined with other re-balancing techniques to improve the performance further. Moreover, our method shows great robustness to a wide range of data distributions, which holds enormous potential in practice. Code will be publicly available.
BibTeX
@misc{chen2022embarrassingly,
  author = {Hao Chen},
  title = {{An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning}},
  year = {2022},
  eprint = {2211.11086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11086},
}
Figure