Manifold Aware Generative Oversampling for Imbalanced Learning

This direction is merged into another direction: majority class bias (class imbalance bias). Its last version stays readable and citable here.

Version 1 of 133 papers36 conceptsfrozen 2026-10-04

In many real-world machine learning tasks, the data is naturally lopsided. Whether detecting rare diseases, identifying credit card fraud, or predicting earthquakes, the "interesting" cases (the minority) are vastly outnumbered by the "normal" cases (the majority). Standard algorithms often fail here, achieving high accuracy by simply ignoring the minority class entirely.

To fix this, researchers have long relied on oversampling: creating synthetic examples of the minority class to balance the scales. However, a new wave of research suggests that the traditional way we build these samples is fundamentally flawed. By moving from simple geometric interpolation to distribution-aware generative modeling, we can create synthetic data that actually helps models generalize.

The Failure of Linear Interpolation

The most common oversampling technique is SMOTE (Synthetic Minority Over-sampling Technique). SMOTE works by picking a minority point, finding its nearest neighbors, and drawing a straight line between them. It then places a new synthetic point somewhere on that line.

As Abdoulaye Sakho, Erwan Scornet, and Emmanuel Malherbe demonstrate in Theoretical and experimental study of SMOTE, this approach has a major geometric limitation: it restricts all synthetic data to the convex hull of the existing minority points.

This "blind" interpolation fails for three main reasons:

  1. Manifold Ignorance: High-dimensional data usually lives on curved, low-dimensional "manifolds." A straight line between two points on a curve often falls off the curve into "empty space" that doesn't represent reality. Ofir Lindenbaum and colleagues argue in Geometry Based Data Generation that density-based models simply replicate the flaws of biased training data.
  2. Lack of Diversity: Because SMOTE only fills the gaps between existing points, it cannot explore the feature space outside the immediate vicinity of observed data. Yan Ma notes in CIGAN that this lack of variance causes models to overfit to the specific training set rather than the real-world distribution.
  3. Noise Generation: In complex datasets, interpolating between two minority points might accidentally place a synthetic sample right in the middle of a majority class cluster, creating "noise" that confuses the decision boundary.

From Lines to Volumes and Subspaces

To move beyond the limitations of 1D line segments, researchers are exploring higher-dimensional geometric structures.

In Simplicial SMOTE, Andrey Savchenko and Gleb Gusev propose sampling from simplices (triangles, tetrahedrons, etc.) rather than just edges. By modeling the minority class as a union of multi-dimensional volumes, the algorithm achieves better data coverage and more effectively pushes the decision boundary away from the minority class.

Another approach, introduced by Kuan-Chih Huang and others in Adaptive Subspace Sampling, uses Self-Organizing Maps to learn the invariant subspaces of the minority data. Instead of guessing where new points should go, the model learns the underlying geometric structure (the "subspace") and generates samples that are mathematically guaranteed to respect that structure.

Generative Modeling as Distribution Learning

The most significant shift in the field is the move toward deep generative models like GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders). Unlike SMOTE, these models don't just interpolate; they attempt to learn the entire probability distribution of the data.

GANs for Refinement and Translation

Several researchers use GANs to "refine" synthetic samples. In SMOGAN, Mike Domaratzki and Shayan Alahyari use a two-stage process: first, they generate samples using traditional oversampling, then they use a GAN to "nudge" those samples until they are statistically indistinguishable from real data.

A more radical approach is seen in TTGAN by Jonathan Gradstein. Instead of creating minority samples from scratch, TTGAN translates majority class samples into the minority class. This ensures the synthetic samples inherit the complex, realistic feature correlations of the majority class while adopting the label of the minority.

VAEs and Latent Space Bootstrapping

VAEs offer a different advantage: they can map complex, messy data into a smooth, organized "latent space." Samuel Stocksieker and colleagues introduce the Deep Smoothed Bootstrap (DSB), which performs oversampling in this learned latent space. Because the latent space is disentangled, the synthetic samples are more likely to respect the underlying physical or logical constraints of the data.

Guiding Generation with Entropy and Uncertainty

If the goal of oversampling is to help a classifier, we shouldn't just generate any minority samples—we should generate the hardest ones.

Jinfu Ren and Jiming Liu 0001 developed EWGAN, which uses an entropy-weighting scheme. By focusing the GAN's attention on samples that the current classifier is most uncertain about, the model generates "informative" synthetic data that specifically clarifies the decision boundary.

Similarly, Amirhossein Zare and Alhassane Zaré propose LEO-CVAE, which uses local Shannon entropy to identify "hard-to-learn" regions. By prioritizing these high-uncertainty areas during synthetic data generation, the model ensures that the new data points are placed exactly where the classifier needs the most help.

The Future: Score-Based Models and LLMs

The frontier of this research direction involves even more powerful generative architectures. S. Park introduced SOS (Score-based OverSampling), the first framework to adapt diffusion-style score-based models for tabular imbalanced data. These models are often more stable than GANs and can capture even more intricate distributions.

Finally, Ryumei Nakada and Linjun Zhang have begun providing a theoretical foundation for using Large Language Models (LLMs) for oversampling. They prove that the "scaling laws" of Transformers allow them to generate high-quality synthetic samples that far outperform SMOTE by capturing the deep, non-linear manifolds of high-dimensional data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_manifold_aware_generative_oversampling_f_7253d3bd,
  title = {Manifold Aware Generative Oversampling for Imbalanced Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manifold-aware-generative-oversampling-for-imbalanced-learning/rdr_7253d3bd8ba84d06a5974f1fd99f1062},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_manifold_aware_generative_oversampling_f_7253d3bd_v1,
  title = {Manifold Aware Generative Oversampling for Imbalanced Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manifold-aware-generative-oversampling-for-imbalanced-learning/rdr_7253d3bd8ba84d06a5974f1fd99f1062/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}