Exploiting Synthetically Generated Data with Semi-Supervised Learning for Small and Imbalanced Datasets

María Pérez-Ortiz, Peter Tiño, Rafal Mantiuk, César Hervás-MartínezAAAI 2019 · DOI

Abstract

Machine learning models often struggle with small or imbalanced datasets, where traditional over-sampling techniques can introduce label noise. This paper proposes a hybrid framework that generates synthetic data points through convex combinations but treats them as unlabeled data within a semi-supervised learning (SSL) context. By utilizing Semi-Supervised Support Vector Machines ($S^3VM$), the model exploits the cluster assumption to find optimal decision boundaries in low-density regions. Extensive evaluations on 53 datasets show that this approach, termed $S^3VM-OvS$, significantly improves robustness and accuracy, especially in high-dimensional spaces and scenarios with significant data loss.

Introduction

The performance of supervised learning is heavily dependent on the availability of large, balanced training sets. In many real-world applications, data is either scarce or highly imbalanced, leading models to overfit or bias toward the majority class. While data augmentation and over-sampling (e.g., SMOTE) are common remedies, they suffer from "label inconsistency." Generating a synthetic point between two examples of the same class assumes the underlying class distribution is convex. If the point falls across a complex decision boundary, the assigned label becomes noise, degrading model performance.

Methods

The proposed framework addresses label noise by decoupling synthetic data generation from labeling.

Synthetic Data Generation

Synthetic points ($x^*$) are created using convex combinations of existing data. For a given seed point, one of its $k$-nearest neighbors is selected, and a new point is generated on the line segment connecting them. This process artificially increases the density of the data clusters.

Conceptual overview of synthetic data generation where new points are created between existing minority class examples.

Semi-Supervised Integration

Instead of naively labeling these synthetic points, they are incorporated into the training set as unlabeled data. The model then employs a Semi-Supervised Support Vector Machine ($S^3VM$), specifically the $S^3VM^{light}$ implementation. This approach treats the labels of synthetic points as optimization variables, allowing the model to determine the most consistent label based on the overall geometry of the feature space. This leverages the "cluster assumption"—the idea that decision boundaries should reside in low-density regions between high-density clusters.

Ensemble Learning

To mitigate the variance introduced by the random selection of neighbors during synthetic generation, the authors use an ensemble of $S^3VM$ models. Final classification is determined via majority voting across the ensemble members.

Key Insights

  • Variance Reduction: Mathematical analysis indicates that convex combinations reduce the variance of the resampled distribution, effectively "thickening" the clusters and making the inductive bias more reliable.
  • Avoiding the Label Trap: By treating synthetic data as unlabeled, the $S^3VM$ can "correct" potentially misplaced synthetic points during optimization, preventing the introduction of label noise near class boundaries.

Experimental Results

The authors evaluated the framework using 27 synthetic datasets (varying in dimensionality, size, and overlap) and 26 real-world benchmarks from the UCI repository.

Comparative performance (boxplots) of the proposed S3VM-OvS (4) against standard SVM (1) and other baselines across different data characteristics.

Performance in High Dimensions

In synthetic tests with $d=100$, the $S^3VM-OvS$ method frequently outperformed models trained on the original, complete dataset. This suggests that the synthetic density helps the classifier navigate the "curse of dimensionality" more effectively.

Robustness to Data Scarcity

To simulate "small data" conditions, the authors removed up to 80% of the data (MCAR 0.8). The proposed ensemble method consistently ranked as the top performer across the benchmark suite, demonstrating superior recovery of the underlying decision boundary compared to standard over-sampling.

Conclusion

The integration of synthetic data generation with semi-supervised learning provides a robust solution for small and imbalanced classification tasks. By treating augmented data as unlabeled, the framework avoids the pitfalls of label noise while benefiting from increased cluster density. Future work may involve extending this approach to multi-class problems and exploring more advanced generative models like VAEs or GANs to produce the unlabeled synthetic samples.

Summary updated

BibTeX
@inproceedings{perezortiz2019exploiting,
  author = {María Pérez-Ortiz and Peter Tiño and Rafal Mantiuk and César Hervás-Martínez},
  title = {{Exploiting Synthetically Generated Data with Semi-Supervised Learning for Small and Imbalanced Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014715},
}
Figure