Robust Generative Modeling for Corrupted Datasets

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 128 papers28 conceptsfrozen 2026-10-04

Standard generative models like Variational Autoencoders (VAEs), GANs, and Diffusion models typically operate under a "clean data" assumption: they treat every sample in a training set as a valid representative of the target distribution. In practice, large-scale datasets are often "dirty," containing sensor errors, mislabeled samples, or web-scraped artifacts. When a model attempts to minimize the distance between its generated distribution and a corrupted training set, it faithfully learns to replicate these outliers, leading to degraded synthesis quality and the generation of nonsensical artifacts.

Recent research has shifted toward Robust Generative Modeling, developing architectures and loss functions that can inherently distinguish between the underlying data manifold and the surrounding noise.

The Fragility of Exact Distribution Matching

The core problem lies in the objective functions used to train these models. Metrics like the Wasserstein distance or standard Variational Inference (VI) are highly sensitive to outliers. In Optimal Transport (OT) frameworks, a single distant outlier can cause transport costs to explode because the model is mathematically forced to account for every bit of probability mass.

As Sloan Nietert and Ziv Goldfeld observe in Outlier-Robust Optimal Transport: Duality, Structure, and Statistical Analysis, standard distance metrics force the generator to model corruption rather than focusing on the clean data manifold. This "garbage in, garbage out" cycle is particularly damaging in diffusion models, where Quan Dao and Anh Tuan Tran show that models trained on uncurated datasets incorporate errors directly into their learned denoising trajectories.

Mechanisms for Robustness

To counter this, researchers have introduced several mechanisms to "ignore" or down-weight suspicious data points during training:

  • Unbalanced and Truncated Transport: Instead of matching the entire distribution, models like UOTM by Jaemoo Choi use "unbalanced" optimal transport. This relaxes the requirement to match every data point, allowing the model to discard outlier mass. Similarly, the ROBOT framework by Debarghya Mukherjee and Justin Solomon caps transport costs at a threshold, effectively neutralizing the impact of distant noise.
  • Risk-Sensitive Diffusion: For tabular and time-series data, Yangming Li and Mihaela van der Schaar propose Risk-Sensitive Diffusion. This method pairs samples with "risk vectors" that indicate noise levels, modifying the underlying stochastic differential equations to mathematically neutralize corruption during the learning process.
  • Heavy-Tailed Priors: In speech enhancement, Ali Golmakani and Mostafa Sadeghi replace standard Gaussian priors in VAEs with Student’s t-distributions. By assigning learnable weights to the variance of different time-frames, the model becomes resilient to non-speech outliers.

Learning the Clean Distribution from Contamination

Beyond just ignoring noise, some approaches actively "purify" the learning signal. PuriGAN, developed by Bowen Tian and Qinliang Su, uses a small auxiliary dataset of "contamination-only" samples to teach the generator what not to model.

Other strategies focus on the geometry of the data. Deli Zhao and Jiapeng Zhu suggest a cluster-based curriculum learning strategy. By training on high-density cluster centers first and only gradually introducing sparser, noisier samples, the model establishes a stable representation of the true data before it can be distorted by outliers.

Why It Works: Intuition

The intuition behind these methods is to move from global fitting to local relevance. Standard models try to explain 100% of the training data. Robust models are designed with the "permission" to explain only 95% of the data perfectly while treating the remaining 5% as irrelevant noise. By introducing thresholds (cost truncation), flexible variances (Student's t-distributions), or risk-awareness, these models ensure that the gradient signal used to update the network is dominated by the structural patterns of the real data rather than the chaotic signals of the outliers.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_generative_modeling_for_corrupted_5be7bff0,
  title = {Robust Generative Modeling for Corrupted Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-generative-modeling-for-corrupted-datasets/rdr_5be7bff00d1d4a2ab600106cd8511943},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_generative_modeling_for_corrupted_5be7bff0_v1,
  title = {Robust Generative Modeling for Corrupted Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-generative-modeling-for-corrupted-datasets/rdr_5be7bff00d1d4a2ab600106cd8511943/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}