SMOTE: Synthetic Minority Over-sampling Technique

Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, W. Philip Kegelmeyer

TL;DR

This paper introduces SMOTE (Synthetic Minority Over-sampling Technique), a novel method for addressing class imbalance in datasets by synthetically generating minority class examples. SMOTE, combined with under-sampling the majority class, significantly improves classifier performance, outperforming traditional over-sampling and other imbalance handling techniques.

What Problem Does This Paper Address?

Many real-world datasets suffer from class imbalance, where one class (the majority class) vastly outnumbers another (the minority class). This is common in areas like fraud detection, medical diagnosis, and anomaly detection. Standard machine learning algorithms, which often optimize for overall accuracy, tend to perform poorly on such datasets. They can become biased towards the majority class, leading to a high rate of misclassification for the critical minority class. Traditional approaches like under-sampling the majority class can lead to loss of valuable information, while simple over-sampling by replication can result in overfitting. This paper tackles the challenge of building classifiers that are sensitive to the minority class without sacrificing performance on the majority class.

Methods

The core contribution of this paper is the SMOTE (Synthetic Minority Over-sampling Technique) algorithm. Unlike traditional over-sampling methods that simply duplicate existing minority class samples, SMOTE creates synthetic minority class samples. Here's how it works:

  1. Identify Nearest Neighbors: For each minority class sample, SMOTE identifies its k nearest neighbors (typically 5).
  2. Generate Synthetic Samples: New synthetic samples are created by interpolating between a minority class sample and one of its randomly chosen nearest neighbors. This involves calculating the difference between the feature vectors of the sample and its neighbor, multiplying this difference by a random number between 0 and 1, and adding it back to the original sample's feature vector.

This process effectively "pushes" the decision boundary of the minority class outwards, making it more general and less prone to overfitting. The authors also propose combining SMOTE with under-sampling the majority class to further enhance classifier performance. They evaluate this combined approach using C4.5, Ripper, and Naive Bayes classifiers.

Datasets

The authors evaluated SMOTE on nine different datasets, varying in size and class imbalance. These include:

  • Pima Indian Diabetes
  • Phoneme
  • Adult
  • E-state
  • Satimage
  • Forest Cover
  • Oil
  • Mammography
  • Can

These datasets represent a diverse range of real-world scenarios with varying degrees of class imbalance, providing a robust testbed for their proposed method.

Key Insights and Ideas

  • Synthetic Generation is Key: The core insight is that creating new, synthetic minority class examples through interpolation is more effective than simply replicating existing ones. Replication leads to overfitting and more specific, smaller decision regions, whereas SMOTE creates broader, more general regions.
  • SMOTE + Under-sampling Synergy: Combining SMOTE (over-sampling minority) with under-sampling the majority class yields superior results compared to either technique alone or other methods like adjusting loss ratios or class priors.
  • ROC Space Dominance: SMOTE-based classifiers consistently dominate in the ROC space, meaning they achieve better trade-offs between true positive and false positive rates across various thresholds.
  • Generalization over Specificity: SMOTE encourages classifiers to learn more general decision regions for the minority class, which is crucial for improving detection rates without excessively increasing false positives.

Key Results

The paper presents extensive experimental results using ROC curves and AUC (Area Under the Curve) as performance metrics.

  • Improved Performance: SMOTE, especially when combined with under-sampling, consistently outperformed plain under-sampling, varying loss ratios in Ripper, and varying class priors in Naive Bayes across most of the tested datasets.
  • ROC Convex Hull: SMOTE-generated classifiers frequently lie on the ROC convex hull, indicating they are potentially optimal.
  • Reduced Tree Size: For decision trees (like C4.5), SMOTE led to smaller tree sizes compared to replication-based over-sampling, suggesting less overfitting.
  • Specific Dataset Performance:
    • For the Pima dataset (least skewed), Naive Bayes performed slightly better than SMOTE-C4.5.
  • For the Oil dataset, Under-Ripper performed slightly better than SMOTE-Ripper.
  • For the Can dataset, SMOTE-classifier and Under-classifier ROC curves overlapped significantly.
  • In all other cases, SMOTE-classifier demonstrated superior performance.

Limitations and Future Work

  • Handling Nominal Features: The original SMOTE algorithm was designed for continuous features. While they propose SMOTE-NC (Nominal Continuous) to handle mixed datasets, its performance on the Adult dataset was not as strong as expected, suggesting further research is needed.
  • Automated Parameter Selection: The number of nearest neighbors (k) and the degree of over-sampling are parameters that need to be set. Future work could explore automated adaptive selection strategies for these parameters.
  • Focus on Misclassified Examples: Investigating strategies for selecting nearest neighbors with a focus on incorrectly classified examples could further improve performance.
  • Application to Information Retrieval: The authors suggest exploring SMOTE's application in information retrieval, which often deals with high-dimensional and imbalanced data.

Summary updated

Abstract

An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of ``normal'' examples with only a small percentage of ``abnormal'' or ``interesting'' examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.
BibTeX
@article{chawla2002smote,
  author = {Nitesh V. Chawla and Kevin W. Bowyer and Lawrence O. Hall and W. Philip Kegelmeyer},
  title = {{SMOTE: Synthetic Minority Over-sampling Technique}},
  journal = {J. Artif. Intell. Res.},
  year = {2002},
  doi = {10.1613/jair.953},
}
Figure