Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning

Kuo Yang, Duo Li, Menghan Hu, Guangtao Zhai, Xiaokang Yang, Xiao-Ping ZhangarXiv 2024 · DOI · arXiv

Introduction

Semi-supervised learning (SSL) has become a cornerstone of modern machine learning, allowing models to leverage vast amounts of unlabeled data. However, a significant challenge remains: real-world data often follows a "long-tailed" distribution, where a few "head" classes dominate while many "tail" classes are underrepresented. Standard SSL techniques like FixMatch often fail in these scenarios due to the "overconfidence trap"—where models generate high-confidence but incorrect pseudo-labels for tail classes—and the "threshold barrier," where fixed high-confidence thresholds exclude tail-class samples entirely.

To address these issues, this paper introduces Uncertainty-Aware Dynamic Threshold Selection (UDTS). UDTS moves beyond simple confidence-based filtering by incorporating model uncertainty and adaptive, class-specific thresholds.

The UDTS Framework

The core of UDTS is a dual-model architecture that integrates uncertainty estimation directly into the pseudo-labeling pipeline.

Uncertainty Estimation via Monte Carlo Dropout

Unlike traditional methods that rely on a single prediction, UDTS employs Monte Carlo (MC) Dropout. By performing multiple forward passes ($T \approx 10$) with dropout enabled, the model generates a distribution of predictions for each unlabeled sample. The variance across these passes serves as a robust measure of uncertainty. Low variance indicates a stable, reliable prediction, while high variance suggests the model is biased or guessing, even if the individual confidence score is high.

Dynamic Threshold Selection

A key innovation of UDTS is the move from a global fixed threshold to class-specific, time-evolving thresholds. Using an Exponential Moving Average (EMA), the framework tracks the learning state of each category.

In the early stages of training, thresholds are kept lower to encourage the model to explore tail-class samples. As the model matures, these thresholds tighten, ensuring that only the most reliable pseudo-labels are incorporated into the training set. This "curriculum" approach prevents the model from being overwhelmed by noise early on while maintaining high precision in later stages.

Experimental Evaluation

The researchers evaluated UDTS on several long-tailed benchmarks, including CIFAR10-LT, CIFAR100-LT, STL-10-LT, and the medical imaging dataset TissueMNIST.

Key Results:

  • CIFAR10-LT: UDTS achieved a significant accuracy boost of over 5.26% compared to the FixMatch baseline.
  • STL-10-LT: The method showed a remarkable 9.96% improvement in accuracy, demonstrating its strength in scenarios with large unlabeled pools.
  • TissueMNIST: An improvement of 1.28% in medical imaging tasks highlights the generalizability of UDTS beyond natural images.
  • Robustness: UDTS consistently outperformed other specialized imbalanced SSL methods like CReST, DARP, and DASO.

Conclusion and Future Work

UDTS effectively mitigates the biases inherent in long-tailed semi-supervised learning by decoupling confidence from reliability. By perceiving its own uncertainty and adapting its selection criteria dynamically, the model achieves superior performance on rare classes without degrading accuracy on common ones.

While the current implementation of MC Dropout introduces computational overhead, future research will focus on more efficient uncertainty estimation techniques, such as single-pass uncertainty heads or knowledge distillation, to scale UDTS to even larger datasets and more complex tasks like object detection.

Summary updated

Abstract

For semi-supervised learning with imbalance classes, the long-tailed distribution of data will increase the model prediction bias toward dominant classes, undermining performance on less frequent classes. Existing methods also face challenges in ensuring the selection of sufficiently reliable pseudo-labels for model training and there is a lack of mechanisms to adjust the selection of more reliable pseudo-labels based on different training stages. To mitigate this issue, we introduce uncertainty into the modeling process for pseudo-label sampling, taking into account that the model performance on the tailed classes varies over different training stages. For example, at the early stage of model training, the limited predictive accuracy of model results in a higher rate of uncertain pseudo-labels. To counter this, we propose an Uncertainty-Aware Dynamic Threshold Selection (UDTS) approach. This approach allows the model to perceive the uncertainty of pseudo-labels at different training stages, thereby adaptively adjusting the selection thresholds for different classes. Compared to other methods such as the baseline method FixMatch, UDTS achieves an increase in accuracy of at least approximately 5.26%, 1.75%, 9.96%, and 1.28% on the natural scene image datasets CIFAR10-LT, CIFAR100-LT, STL-10-LT, and the medical image dataset TissueMNIST, respectively. The source code of UDTS is publicly available at: https://github.com/yangk/UDTS.
BibTeX
@misc{yang2024uncertainty,
  author = {Kuo Yang and Duo Li and Menghan Hu and Guangtao Zhai and Xiaokang Yang and Xiao-Ping Zhang},
  title = {{Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning}},
  year = {2024},
  eprint = {2401.04435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.04435},
}
Figure