A systematic study of the class imbalance problem in convolutional neural networks

Mateusz Buda, Atsuto Maki, Maciej A. Mazurowski

Abstract

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

Introduction

Class imbalance occurs when certain classes in a dataset significantly outnumber others. This is a common challenge in real-world applications, such as medical imaging or fraud detection, where the "interesting" cases are often rare. Standard training algorithms are designed to maximize overall accuracy, which can lead to models that perform well on majority classes but fail to learn the characteristics of minority classes. While traditional machine learning has established strategies for this problem, deep learning has often relied on intuition. This paper by Buda, Maki, and Mazurowski provides a systematic evaluation of how class imbalance affects Convolutional Neural Networks (CNNs).

Methods and Experimental Setup

The researchers evaluated seven strategies to mitigate imbalance, including random minority oversampling, random majority undersampling, two-phase training, and thresholding (adjusting output probabilities during inference based on class priors). The study specifically examined two types of imbalance distributions: Step Imbalance, where classes are divided into large and small groups, and Linear Imbalance, where sample counts decrease smoothly across classes.

Imbalance distributions

Experiments were conducted across three benchmark datasets of increasing complexity to ensure the findings scaled appropriately: MNIST (simple), CIFAR-10 (medium), and ImageNet (high complexity). These datasets represent a range of image dimensions and classification challenges.

Experimental setup

Key Results and Discussion

The primary evaluation metric was Multi-class ROC AUC, which measures ranking performance independent of class prevalence.

The Dominance of Oversampling

Oversampling emerged as the most robust solution across nearly all scenarios. Contrary to common belief in classical machine learning—where oversampling is often avoided due to the risk of overfitting—CNNs proved remarkably resistant. Replicating minority samples consistently improved generalization on the test set. For maximum effectiveness, the authors recommend oversampling until classes are perfectly balanced.

Oversampling vs. Undersampling

Oversampling consistently outperformed undersampling. Undersampling discards data, which appears to hinder the ability of CNNs to learn the complex, high-dimensional features necessary for high-quality classification. While the optimal undersampling ratio varies with the degree of imbalance, oversampling remains a more stable choice.

Impact of Task Complexity

The detrimental effect of class imbalance scales with task complexity. While MNIST showed relative resilience, performance on ImageNet dropped significantly as imbalance increased. For instance, baseline ROC AUC dropped from ~99% under low imbalance to ~90% under extreme imbalance conditions.

Conclusion

This systematic study clarifies the impact of class imbalance on CNNs. The findings suggest that oversampling is a highly effective and safe method for addressing imbalance in deep learning, debunking the myth that it necessarily leads to overfitting. While thresholding is useful for improving overall accuracy, oversampling remains the dominant technique for improving the model's fundamental discriminative ability. Future work could explore more sophisticated sampling methods like SMOTE to see if they offer further advantages over simple replication.

Summary updated

Abstract

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.
BibTeX
@inproceedings{buda2018systematic,
  author = {Mateusz Buda and Atsuto Maki and Maciej A. Mazurowski},
  title = {{A systematic study of the class imbalance problem in convolutional neural networks}},
  booktitle = {Neural Networks},
  year = {2018},
  url = {https://openreview.net/forum?id=2m6fqjb11i},
}
Figure