Dynamic Re-weighting for Long-tailed Semi-supervised Learning

Hanyu Peng, Weiguo Pian, Mingming Sun, Ping LiWACV 2023 · OpenReview

Abstract

Semi-supervised Learning (SSL) reduces significant human annotations by simply demanding a small number of labelled samples and a large number of unlabelled samples. The research community has often developed SSL regarding the nature of a balanced data set; in contrast, real data is often imbalanced or even long-tailed. The need to study SSL under imbalance is therefore critical. In this paper, we essentially extend FixMatch (a SSL method) to the imbalanced case. We find that the unlabeled data is as well highly imbalanced during the training process; in this respect we propose a re-weighting solution based on the effective number. Furthermore, since prediction uncertainty leads to temporal variations in the number of pseudo-labels, we are innovative in proposing a dynamic reweighting scheme on the unlabeled data. The simplicity and validity of our method are backed up by experimental evidence. Especially on CIFAR-10, CIFAR-100, ImageNet127 data sets, our approach provides the strongest results against previous methods across various scales of imbalance.

Introduction

In practical machine learning applications, datasets rarely adhere to the ideal of class balance. Real-world data—from medical imaging to e-commerce—is typically long-tailed, where a few "head" classes dominate the sample count while many "tail" classes are underrepresented. While Semi-supervised Learning (SSL) has emerged as a powerful paradigm for learning from limited labeled data, standard algorithms like FixMatch are primarily designed for balanced distributions.

When applied to long-tailed data, these algorithms often fail because the model becomes biased toward majority classes. This bias is exacerbated in SSL through the pseudo-labeling process: the model predicts head classes more frequently, generating more majority-class pseudo-labels and further marginalizing the tail. This paper introduces Dynamic Re-weighting (DRw), a scheme designed to address both the inherent imbalance in unlabeled data and the temporal variations in pseudo-label distribution during training.

Methodology

The proposed DRw method extends the FixMatch framework, which utilizes consistency regularization between weakly and strongly augmented versions of unlabeled images. The core innovation lies in how the loss function is weighted to account for class imbalance.

Effective Number and Re-weighting

Instead of relying on raw sample counts, the authors utilize the concept of the effective number ($E_n$). This metric accounts for data redundancy; as the number of samples in a class increases, the marginal benefit of each new sample decreases.

The relationship between the number of samples and the effective number for different values of hyperparameter β.

The effective number is defined as $E_n = (1 - \beta^n) / (1 - \beta)$, where $n$ is the number of samples and $\beta \in [0, 1)$ is a hyperparameter. In the context of SSL, where true labels for unlabeled data are unknown, the authors track "pseudo-numbers"—the count of pseudo-labels assigned to each class by the model.

Dynamic Scaling

Unlike supervised learning where class counts are static, the distribution of pseudo-labels in SSL is a "moving target" that evolves as the model learns. DRw is "dynamic" because it updates these pseudo-numbers every epoch. The loss function is then re-weighted such that classes with a lower effective number of pseudo-labels receive higher weights, forcing the model to focus on the underrepresented tail classes.

Key Insights and Experimental Analysis

A critical finding of this research is that the model's total loss on unlabeled data is extremely imbalanced. Visualization of the training process reveals that the model over-optimizes for head classes while tail classes remain under-trained.

Imbalance in loss values across different class indices, highlighting the bias toward majority classes.

By implementing dynamic re-weighting, the model compensates for this disparity. The simplicity of DRw is a key advantage; it does not require complex auxiliary models or distribution alignment, but rather a more intelligent weighting of the existing loss function based on real-time model predictions.

Results

The authors evaluated DRw on CIFAR-10, CIFAR-100, and ImageNet127, simulating various imbalance ratios (the ratio of the most frequent to least frequent class).

  • CIFAR-10: At an extreme imbalance ratio of 150, DRw achieved 74.1% accuracy, outperforming the FixMatch + CReST baseline by 1.3%.
  • CIFAR-100: On a ratio of 20, it reached 57.1%, a 2% improvement over competing methods.
  • ImageNet127: On this large-scale benchmark, DRw achieved 75.2% accuracy, significantly outperforming standard FixMatch (65.8%) and CReST+ (73.3%).
  • Synergy: When combined with Logit Adjustment (LA), performance further improved, reaching 86.5% on CIFAR-10 with an imbalance ratio of 50.

Conclusion

Dynamic Re-weighting (DRw) provides a robust and elegant solution for long-tailed semi-supervised learning. By adapting to the evolving distribution of pseudo-labels, it ensures that minority classes are not ignored during the training process. While the method shows sensitivity to batch size and requires tuning of the $\beta$ hyperparameter, its performance across diverse benchmarks demonstrates its viability for real-world, imbalanced applications.

Summary updated

Abstract

Semi-supervised Learning (SSL) reduces significant human annotations by simply demanding a small number of labelled samples and a large number of unlabelled samples. The research community has often developed SSL regarding the nature of a balanced data set; in contrast, real data is often imbalanced or even long-tailed. The need to study SSL under imbalance is therefore critical. In this paper, we essentially extend FixMatch (a SSL method) to the imbalanced case. We find that the unlabeled data is as well highly imbalanced during the training process; in this respect we propose a re-weighting solution based on the effective number. Furthermore, since prediction uncertainty leads to temporal variations in the number of pseudo-labels, we are innovative in proposing a dynamic reweighting scheme on the unlabeled data. The simplicity and validity of our method are backed up by experimental evidence. Especially on CIFAR-10, CIFAR-100, ImageNet127 data sets, our approach provides the strongest results against previous methods across various scales of imbalance.
BibTeX
@inproceedings{peng2023dynamic,
  author = {Hanyu Peng and Weiguo Pian and Mingming Sun and Ping Li},
  title = {{Dynamic Re-weighting for Long-tailed Semi-supervised Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=O6vJ9KjU2H8},
}
Figure