Abstract
A key challenge of oversampling in imbalanced classification is that the generation of new minority samples often neglects the usage of majority classes, resulting in most new minority sampling spreading the whole minority space. In view of this, we present a new oversampling framework based on the counterfactual theory. Our framework introduces a counterfactual objective by leveraging the rich inherent information of majority classes and explicitly perturbing majority samples to generate new samples in the territory of minority space. It can be analytically shown that the new minority samples satisfy the minimum inversion, and therefore most of them locate near the decision boundary. Empirical evaluations on benchmark datasets suggest that our approach significantly outperforms the state-of-the-art methods.
Introduction
Imbalanced classification occurs when one class (the majority) vastly outnumbers another (the minority), a common scenario in fields like fraud detection and medical diagnosis. Standard machine learning algorithms often struggle in these settings, as they tend to favor the majority class to maximize overall accuracy. Oversampling techniques, such as SMOTE, attempt to rebalance the data by generating synthetic minority samples. However, these methods typically focus exclusively on existing minority samples, ignoring the valuable information contained within the majority class distribution. This often leads to "boundary blurring," where synthetic samples overlap with the majority space, creating noise that confuses the classifier.
Methodology: The Counterfactual Framework
The proposed framework addresses these limitations by utilizing counterfactual theory. Instead of interpolating between minority points, the method generates synthetic samples by perturbing majority samples until they are classified as the minority class.
The Counterfactual Objective and Minimum Inversion
The core idea is to find a synthetic sample $x'$ that is as close as possible to a factual majority sample $x$, but which a classifier $f_w$ labels as the minority class. Formally, this is achieved by minimizing the distance $d(x, x')$ subject to $f_w(x') = \text{minority}$. By minimizing this distance, the algorithm ensures minimum inversion, which naturally places the new samples near the decision boundary.
Implementation Details
- Truncated Normal Perturbation: To maintain the realism of the synthetic data, features are "nudged" using a truncated normal distribution, ensuring they stay within the observed range of the original data.
- Feature Ranking: For high-dimensional datasets, Spearman correlation is used to rank features by their impact on the classification. The algorithm prioritizes perturbing these key features, significantly reducing computational complexity.
Experimental Results
The framework was evaluated on five diverse datasets, ranging from synthetic 2D data to high-dimensional medical records (120 features).
As illustrated in Figure 2, traditional oversampling methods often generate samples that "bleed" into the majority space or fill the interior of the minority cluster. In contrast, the counterfactual method (panel d) generates samples that are strictly within the minority territory and concentrated along the decision boundary.
Quantitatively, the method showed significant improvements in F-measure and G-Mean across all datasets. Notably, on the Page Blocks Classification (PBC) dataset, the approach yielded a 15% performance boost compared to standard undersampling methods. The results were consistent across various classifiers (SVM, Random Forest, GBDT), demonstrating the robustness of the generated balanced training sets.
Conclusion
By leveraging the majority class as a guide, counterfactual-based oversampling produces high-quality synthetic data that emphasizes the critical decision boundary. This approach avoids the noise common in minority-only methods and provides a more effective training signal for classifiers. Future work will focus on integrating the sample generation and classifier training into a unified end-to-end framework.
Summary updated