A systematic study of the class imbalance problem in convolutional neural networks

Mateusz Buda, Atsuto Maki, Maciej A. Mazurowski

In the world of machine learning, we often dream of perfectly balanced datasets where every category is represented equally. In reality, data is messy. Whether you are detecting rare diseases in medical images or spotting fraudulent transactions, the "interesting" cases are often outnumbered by the "normal" ones by a factor of 1,000 to 1.

While traditional machine learning has a well-established playbook for this "class imbalance problem," deep learning has long relied on intuition and ad-hoc fixes. This paper by Buda, Maki, and Mazurowski provides the systematic rigour the field was missing.

TL;DR

This study rigorously compares methods for handling class imbalance in Convolutional Neural Networks (CNNs) across datasets of varying complexity. The core takeaway: oversampling is the most robust solution and, surprisingly, does not cause the overfitting typically seen in classical models.

What Problem Does This Paper Address?

Class imbalance occurs when some classes have significantly more examples than others. This is a "silent killer" for model performance because standard training algorithms are designed to maximize overall accuracy. If 99% of your data is "Healthy," a model can achieve 99% accuracy just by guessing "Healthy" every time—effectively learning nothing about the 1% of "Sick" cases.

While researchers have used various tricks to fix this, there hasn't been a definitive guide on which methods work best for deep learning architectures like CNNs. The authors set out to determine if the rules of thumb from "classical" ML (like Random Forests or SVMs) still apply to the high-dimensional world of deep learning.

Methods

The authors tested seven different strategies to combat imbalance:

  1. Random Minority Oversampling: Replicating random samples from the minority classes.
  2. Random Majority Undersampling: Throwing away random samples from the majority classes.
  3. Two-Phase Training: Pre-training on a balanced version of the data (using oversampling or undersampling) and then fine-tuning the output layers on the original imbalanced data.
  4. Thresholding: Adjusting the model's output probabilities during inference to compensate for the prior probability of each class.
  5. Hybrid Approaches: Combining oversampling/undersampling with thresholding.

They also defined two specific types of imbalance to test:

  • Step Imbalance: A sharp "step" where some classes are large and others are small.
  • Linear Imbalance: A smooth, linear decrease in the number of samples across classes.

Datasets

The researchers used three benchmark datasets of increasing complexity to see how their findings scaled:

  • MNIST: Grayscale digits (simple).
  • CIFAR-10: Small color images of objects/animals (medium complexity).
  • ImageNet (ILSVRC-2012): Large-scale, high-resolution images (high complexity).

By using these three, they could observe if a technique that works on simple digits still holds up when classifying 1,000 different categories of real-world objects.

Key Insights and Ideas

  • The Overfitting Myth: In classical ML, oversampling is often criticized because it can lead to overfitting (the model just memorizes the specific replicated samples). The authors found that CNNs are remarkably resistant to this. Oversampling consistently improved generalization on the test set.
  • Oversampling > Undersampling: In almost every scenario, oversampling outperformed undersampling. Undersampling discards potentially valuable data, which hurts the deep features CNNs need to learn.
  • The "Total Elimination" Rule: For oversampling to be most effective, you should oversample until the classes are perfectly balanced. For undersampling, however, the "optimal" ratio depends on how extreme the initial imbalance was.
  • Complexity Matters: The negative impact of class imbalance is significantly stronger as the task becomes more complex. A level of imbalance that barely nudges the needle on MNIST can cause a massive performance drop on ImageNet.

Key Results

The authors primarily used Multi-class ROC AUC as their metric, as it measures the model's ability to rank classes correctly regardless of their prevalence.

  • Dominance of Oversampling: On MNIST and CIFAR-10, oversampling (and its two-phase variant) consistently sat at the top of the leaderboard.
  • ImageNet Performance: On ImageNet, the baseline ROC AUC dropped from ~99% (low imbalance) to ~90% (extreme imbalance). Even here, oversampling remained the most consistent performer.
  • Thresholding for Accuracy: While thresholding doesn't change the ROC AUC (since it's a post-processing step), the authors found it is the best way to improve overall accuracy if that is your primary goal, especially when combined with oversampling.

Limitations and Future Work

The authors noted that their ImageNet experiments were computationally expensive (taking five days per run on a GTX 1070), which limited the number of trials they could perform. They cautioned that in extremely high-complexity tasks with massive imbalance, the "best" method might require even more specialized study.

Additionally, they focused on "random" sampling. Future work could explore more "informed" sampling methods (like SMOTE or synthetic data generation) to see if they offer any further edge over the simple replication used here.

Summary updated

Abstract

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.
BibTeX
@inproceedings{buda2018systematic,
  author = {Mateusz Buda and Atsuto Maki and Maciej A. Mazurowski},
  title = {{A systematic study of the class imbalance problem in convolutional neural networks}},
  booktitle = {Neural Networks},
  year = {2018},
  url = {https://openreview.net/forum?id=2m6fqjb11i},
}

Loading related content...

Figure