CSMOUTE: Combined Synthetic Oversampling and Undersampling Technique for Imbalanced Data Classification

Michal KoziarskiIJCNN 2021 · DOI

TL;DR

This paper introduces CSMOUTE, a novel data-level technique that combines synthetic oversampling (SMOTE) with a new synthetic undersampling method (SMUTE) to tackle imbalanced datasets. CSMOUTE shows promise in improving classification performance, particularly on datasets with many outliers, and offers a more nuanced approach than traditional over- or undersampling alone.

What Problem Does This Paper Address?

Many real-world classification problems suffer from data imbalance, where one class (the majority class) has significantly more instances than others (minority classes). This imbalance can lead traditional machine learning algorithms to become biased towards the majority class, resulting in poor performance on the minority classes, which are often the most critical (e.g., fraud detection, medical diagnoses). Existing solutions fall into two categories: data-level methods that manipulate the training data (oversampling or undersampling) and algorithm-level methods that adjust the learning algorithm itself. This paper focuses on data-level approaches, aiming to improve upon existing techniques by combining the strengths of both oversampling and undersampling.

Methods

The paper proposes two main techniques:

  • SMUTE (Synthetic Majority Undersampling Technique): This is a novel undersampling method that draws inspiration from SMOTE's interpolation concept. Instead of simply removing random majority class instances, SMUTE synthesizes new majority class instances by interpolating between a randomly selected majority instance and one of its nearest neighbors. This process aims to reduce the majority class while retaining more informative instances.
  • CSMOUTE (Combined Synthetic Oversampling and Undersampling Technique): This method combines the proposed SMUTE with the well-established SMOTE. CSMOUTE allows for a tunable ratio, enabling users to control the balance between oversampling minority instances and undersampling majority instances. This hybrid approach aims to leverage the benefits of both strategies simultaneously.

Datasets

The experimental study was conducted on binary imbalanced datasets from the KEEL repository. These datasets vary in imbalance ratio, number of samples, and features. No new benchmark datasets were introduced.

Key Insights and Ideas

  • SMUTE offers a more sophisticated undersampling: By interpolating majority class instances, SMUTE aims to create more representative "synthetic" majority points, potentially preserving more valuable information than random undersampling.
  • Combining over- and undersampling can be beneficial: CSMOUTE demonstrates that a hybrid approach can outperform either oversampling or undersampling alone, especially for certain classifiers and performance metrics.
  • Dataset characteristics matter significantly: The performance of CSMOUTE is strongly influenced by the types of minority class instances present. It performs better on datasets with a high proportion of outliers (suggesting robustness to noise) but tends to perform worse on datasets with many borderline instances.
  • Classifier choice is crucial: The effectiveness of CSMOUTE varies with the classifier. It shows more significant improvements with classifiers like MLP and SVM, which often handle more complex decision boundaries, compared to logistic regression.

Key Results

  • SMUTE vs. Random Undersampling (RUS): SMUTE achieved better performance than RUS on a higher number of datasets, particularly when using the F-measure metric, showing statistical significance.
  • CSMOUTE's Ratio Impact: For classifiers like SVM and MLP, and metrics like AUC and G-mean, CSMOUTE showed optimal performance at small ratio values (meaning more undersampling than oversampling). MLP, in particular, achieved the best average performance when using CSMOUTE.
  • CSMOUTE vs. Reference Methods: CSMOUTE achieved favorable average ranks compared to individual over- and undersampling methods. While it didn't consistently outperform combined over/undersampling methods like SMOTE+TL and SMOTE+ENN, it also didn't perform significantly worse than any method.
  • Correlation with Dataset Characteristics: A statistically significant positive correlation was found between CSMOUTE's performance (AUC metric) and the proportion of outlier instances. Conversely, a negative correlation was observed with borderline instances, indicating worse performance in their presence.

Limitations and Future Work

  • Ad-hoc nature of SMUTE: The paper acknowledges that SMUTE, like SMOTE, doesn't explicitly consider the positions of majority class objects, potentially leading to synthetic instances overlapping the majority class distribution.
  • Need for further analysis: The authors suggest that while the proposed approach shows promise, further experimental analysis is needed to confirm hypotheses about its performance with classifiers that create more complex decision boundaries.
  • Selective resampling: The findings on dataset characteristics suggest that a promising future direction is selective resampling, where the approach might focus on outlier instances during undersampling and treat borderline instances differently (e.g., by excluding them or applying oversampling). This would involve adapting the resampling strategy based on local data characteristics.

Summary updated

BibTeX
@inproceedings{koziarski2021csmoute,
  author = {Michal Koziarski},
  title = {{CSMOUTE: Combined Synthetic Oversampling and Undersampling Technique for Imbalanced Data Classification}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533415},
}
Figure