SMOTE: Synthetic Minority Over-sampling Technique

Kevin W. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, W. Philip Kegelmeyer

Abstract: An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of "normal" examples with only a small percentage of "abnormal" or "interesting" examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.

Introduction: The Class Imbalance Problem

In many real-world scenarios—such as detecting credit card fraud, identifying cancerous cells, or spotting oil spills in satellite images—the "interesting" class is extremely rare. Standard machine learning algorithms are often biased toward the majority class because they aim to minimize global error. If only 1% of data is "abnormal," a classifier can achieve 99% accuracy just by predicting "normal" every time—a phenomenon known as the accuracy trap.

Visualization of the Mammography dataset across two attributes, illustrating the extreme class imbalance between the majority (green) and minority (red) classes.

Previous attempts to fix this involved either under-sampling the majority (throwing away potentially useful data) or over-sampling the minority by duplicating examples. However, simple replication often causes the model to "memorize" specific points (overfitting) rather than learning general patterns. The authors sought a way to bolster the minority class without the pitfalls of simple replication.

Methodology: Synthetic Minority Over-sampling Technique (SMOTE)

The core innovation of this paper is SMOTE, which operates in the feature space rather than the data space. Instead of duplicating existing points, SMOTE creates entirely new, "synthetic" examples by interpolating between existing minority points.

For each minority class sample, the algorithm identifies its $k$-nearest neighbors (typically $k=5$). It then selects one neighbor at random and creates a new point along the line segment connecting them using the formula: $$x_{new} = x_i + \text{rand}(0,1) \times (x_{neighbor} - x_i)$$

This approach "spreads" the minority class decision region further into the majority class space. This forces classifiers, such as decision trees, to learn broader and more generalizable rules rather than highly specific, overfitted ones.

Experimental Setup

The authors evaluated SMOTE across nine diverse datasets with varying degrees of imbalance, ranging from 2:1 to over 100:1:

  • Medical: Pima Indian Diabetes, Mammography (identifying calcifications).
  • Environment/Science: Oil spill detection, Forest Cover, Satimage, E-state.
  • General/Social: Adult census data, Phoneme recognition.
  • Industrial: Structural engineering simulation (Can).

Performance Metrics and ROC Analysis

The paper reinforces that accuracy is an insufficient metric for imbalanced data. Instead, the authors rely on Receiver Operating Characteristic (ROC) curves and the Area Under the Curve (AUC). These metrics provide a better understanding of the trade-off between the True Positive rate (sensitivity) and the False Positive rate.

ROC curve illustrating how increased undersampling of the majority class moves the operating point toward the ideal point (100% True Positive, 0% False Positive).

Key Results and Discussion

The most effective strategy identified was a combination of SMOTE-ing the minority and under-sampling the majority. This balanced the dataset while providing the model with a richer variety of minority examples to learn from.

  • Dominance in ROC Space: In almost every dataset, the SMOTE + Under-sampling approach outperformed other methods, such as varying loss ratios in Ripper or class priors in Naive Bayes, across the ROC convex hull.
  • Tree Complexity: Classifiers trained on SMOTE data were generally more efficient. For the Mammography dataset, SMOTE achieved better minority recognition with a smaller, more general tree compared to simple replication, which produced a bloated, overfitted tree.
  • Consistency: Out of 48 experimental configurations, the SMOTE-based classifier was the top performer in 44 cases.

Conclusion and Future Work

While SMOTE is highly effective, it has limitations. The basic algorithm assumes continuous features; the authors introduced SMOTE-NC (Nominal Continuous) to handle mixed data types. Additionally, if a minority sample is surrounded by majority samples (noise), SMOTE might generate synthetic points that reinforce that noise. Future work could focus on "intelligent" neighbor selection to avoid over-sampling noisy regions. Despite these challenges, SMOTE remains a foundational technique for addressing class imbalance in modern machine learning.

Summary updated

Abstract

An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of "normal" examples with only a small percentage of "abnormal" or "interesting" examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.
BibTeX
@article{bowyer2002smote,
  author = {Kevin W. Bowyer and Nitesh V. Chawla and Lawrence O. Hall and W. Philip Kegelmeyer},
  title = {{SMOTE: Synthetic Minority Over-sampling Technique}},
  journal = {J. Artif. Intell. Res.},
  year = {2002},
  doi = {10.1613/jair.953},
}
Figure