Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding

Lan-Zhe Guo, Yu-Feng LiICML 2022 · OpenReview

Overview

Standard semi-supervised learning (SSL) algorithms often fail on real-world, imbalanced datasets because they use a fixed confidence threshold that inadvertently ignores minority classes. To address this, researchers have introduced Adsh (Adaptive Thresholding), a framework that dynamically adjusts thresholds for each class. This ensures that minority classes receive high-quality pseudo-labels, significantly boosting overall model performance.

The Challenge of Imbalanced Data

Semi-supervised learning is a powerful tool for training models when labeled data is scarce but unlabeled data is plentiful. However, most popular SSL algorithms, such as FixMatch, rely on a "fixed threshold" strategy. They only generate a pseudo-label for an unlabeled image if the model's confidence exceeds a high bar (e.g., 0.95).

In the real world, data is rarely balanced. When classes are imbalanced, models naturally become more confident in "majority" classes and less confident in "minority" ones. A fixed threshold of 0.95 might capture thousands of majority-class samples but zero minority-class samples. This creates a feedback loop where the model never learns the "tail" of the distribution, leading to poor recall on rare categories.

Imbalanced dataset distribution and the resulting test recall improvements using Adsh compared to FixMatch.

The Adsh Framework

The authors propose Adsh (Adaptive Thresholding) to break this cycle by making the threshold class-dependent and adaptive. The core innovation is moving from a single global threshold to a set of class-specific thresholds.

  1. Optimization Formulation: The standard SSL objective function is reformulated to explicitly include the number of pseudo-labels selected for each class. By introducing a class-bias term, the authors mathematically derive an optimal way to select pseudo-labels.
  2. Closed-Form Solution: They prove that the optimal pseudo-labeling strategy has a closed-form solution. Instead of just checking if a prediction exceeds a static threshold, the model checks if the prediction—adjusted by the class bias—is high enough.
  3. The Adsh Algorithm: To implement this without needing to know the true, often unknown, distribution of the unlabeled data, Adsh uses a ranking strategy:
  4. It ranks all unlabeled predictions for each class by confidence.
  5. It determines what percentage of the majority class's samples pass a standard high-confidence threshold (e.g., 0.95).
  6. It applies that same percentage to all other classes. For example, if the top 80% of majority class predictions are confident enough, the algorithm will also take the top 80% of minority class predictions, even if their absolute confidence scores are lower.

Experimental Setup and Key Insights

The researchers evaluated Adsh on long-tailed (imbalanced) versions of three standard computer vision benchmarks: CIFAR-10, SVHN (Street View House Numbers), and STL-10. They tested these across various imbalance ratios ($\gamma$) ranging from 50 to 150, where the most frequent class has up to 150 times more samples than the rarest.

Relative Confidence Matters: The critical insight is that absolute confidence is a biased metric in imbalanced settings. A model's 0.70 confidence in a rare class might be just as "correct" as its 0.95 confidence in a common class.

Distribution Mismatch Robustness: Adsh performs remarkably well even when the imbalance in the labeled data doesn't match the imbalance in the unlabeled data—a common and difficult scenario in real-world applications.

Key Results

Adsh consistently outperformed existing methods across various imbalance settings and labeled data counts.

Accuracy comparison across different unlabeled imbalance ratios, showing Adsh outperforming DARP and FixMatch.
  • Superior Accuracy: On imbalanced CIFAR-10 with an imbalance ratio of 150, Adsh achieved 71.49% accuracy, outperforming the previous state-of-the-art, CReST (69.84%), and standard FixMatch (68.47%).
  • Cleaner Confusion Matrices: Adsh significantly reduces the tendency of the model to misclassify minority samples as majority ones compared to FixMatch.
  • Synergy with Other Methods: When combined with "Classifier Re-Training" (cRT), a common technique for imbalanced learning, Adsh reached even higher performance, achieving 86.21% on CIFAR-10 with an imbalance ratio of 50.

Limitations and Future Work

While Adsh is empirically very strong, the authors acknowledge that it currently lacks theoretical guarantees, such as mathematical proofs that it will always converge to the best possible solution. Future work could involve providing a formal convergence analysis, refining the initial threshold selection to be more automated, and exploring how adaptive thresholding could be applied to other domains like Natural Language Processing (NLP) or medical imaging, where class imbalance is rampant.

Summary updated

Abstract

Semi-supervised learning (SSL) has proven to be successful in overcoming labeling difficulties by leveraging unlabeled data. Previous SSL algorithms typically assume a balanced class distribution. ...
BibTeX
@inproceedings{guo2022class,
  author = {Lan-Zhe Guo and Yu-Feng Li},
  title = {{Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=uKsh01T_ZFW},
}
Figure