Abstract
In practice, machine learning experts are often confronted with imbalanced data. Without accounting for the imbalance, common classifiers perform poorly and standard evaluation metrics mislead the practitioners on the model's performance. A common method to treat imbalanced datasets is under- and oversampling. In this process, samples are either removed from the majority class or synthetic samples are added to the minority class. In this paper, we follow up on recent developments in deep learning. We take proposals of deep generative models, including our own, and study the ability of these approaches to provide realistic samples that improve performance on imbalanced classification tasks via oversampling. Across 160K+ experiments, we show that all of the new methods tend to perform better than simple baseline methods such as SMOTE, but require different under- and oversampling ratios to do so. Our experiments show that the way the method of sampling does not affect quality, but runtime varies widely. We also observe that the improvements in terms of performance metric, while shown to be significant when ranking the methods, often are minor in absolute terms, especially compared to the required effort. Furthermore, we notice that a large part of the improvement is due to undersampling, not oversampling. We make our code and testing framework available.
Introduction
Real-world tabular datasets, such as those used for fraud detection or medical diagnosis, are frequently imbalanced. In these scenarios, the minority class—the "interesting" cases—is often significantly outnumbered by the majority class. Standard machine learning classifiers, optimized for overall accuracy, often fail to capture the nuances of the minority class. Traditional resampling techniques like SMOTE (Synthetic Minority Over-sampling Technique) have been the standard for years, but they often struggle with the complex, non-linear relationships inherent in mixed tabular data. This study investigates whether Deep Generative Models (DGMs) can provide a more robust solution for generating realistic synthetic samples.
Methodology
The researchers evaluated several DGM architectures, including Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) like MedGAN, ARAE, and WGAN-GP. A critical challenge in tabular data is the presence of both numerical and categorical variables. To address this, the authors developed Multi-Variable (MV) extensions that utilize Gumbel-Softmax activations for categorical outputs and specialized reconstruction losses to handle mixed data types effectively.
The study compared three generation strategies: training only on the minority class, training conditionally on all data with class labels, and rejection sampling. Experiments were conducted on three benchmark datasets: Adult (income prediction), Default of Credit Card Clients, and Credit Card Fraud.
Results and Analysis
Across more than 160,000 experiments, the study revealed that while DGMs can achieve statistically significant improvements over traditional baselines, the absolute gains are often marginal. For instance, on the Adult dataset, the best DGM (MV-WGAN) reached an F1-score of 0.734 compared to SMOTE's 0.732.
A pivotal finding is the sensitivity of model performance to the Undersampling Ratio (USR) and Oversampling Ratio (OSR). The "boost" in performance often attributed to sophisticated oversampling is frequently driven by the simultaneous undersampling of the majority class. The heatmaps below illustrate how performance varies across these ratios for different methods.
Interestingly, on the extremely imbalanced Credit Card Fraud dataset, SMOTE outperformed the deep learning models (0.831 vs. 0.822 F1). This suggests that when data is extremely sparse, simpler geometric interpolation may be more reliable than complex neural network training. Furthermore, modern gradient-boosted trees like XGBoost are already quite robust to imbalance, leaving less room for improvement through resampling.
Conclusion
While Deep Generative Models offer a sophisticated approach to oversampling, their practical utility is tempered by high computational costs and extreme sensitivity to hyperparameter tuning, specifically the balance between under- and oversampling ratios. For many practitioners, the marginal performance gains may not justify the increased complexity and runtime compared to established methods like SMOTE or robust classifiers like XGBoost. Future research should focus on automating the discovery of optimal sampling ratios to make these advanced models more accessible.
Summary updated