Generative Data Augmentation Without External Data: An Empirical and Theoretical Study on Deep Networks

Daniel Kua, Yan SongarXiv 2026 · arXiv

TL;DR

This paper investigates whether generative data augmentation (GDA) actually works when you can't "cheat" by using pre-trained models like Stable Diffusion. By training generators and classifiers from scratch on the same data, the authors show that GDA is a double-edged sword: it only helps when the generator's distribution shift is minimal and the classifier has significant room to improve, often failing to outperform traditional, cheaper augmentation techniques.

What Problem Does This Paper Address?

Generative Data Augmentation (GDA)—adding synthetic images to a training set—is often hailed as a solution for data scarcity. However, most successful applications use generators pre-trained on massive external datasets (e.g., LAION). This makes it unclear if the gains come from the augmentation process itself or simply from knowledge transfer from the pre-training data.

The authors isolate the impact of GDA by studying the "no external data" regime. They ask: If a generator only sees the same training data as the classifier, when and why does adding synthetic samples help? This is critical for domains like medical imaging, where privacy and licensing often prevent the use of external datasets.

Methods

The authors conduct a massive empirical sweep combined with a new theoretical framework:

  1. The Pipeline: They train a generator from scratch on a fraction of the real data, sample synthetic images, mix them with real ones, and train a classifier from scratch on this mixture.
  2. Generative Models: They compare four distinct types: Variational Autoencoders (VAEs), Denoising Diffusion Probabilistic Models (DDPMs), Score-based models (VP-SDE), and Flow Matching (FM).
  3. Classifier Backbones: They test four architectures to ensure results aren't backbone-specific: ResNet-18, Vision Transformer (ViT), Swin Transformer, and ConvNeXt-V2.
  4. Theoretical Bound: They derive an "excess risk bound" that quantifies the trade-off between the statistical benefit of a larger training set and the cost of training on a shifted distribution (measured via Total Variation distance).
Figure 3. The generator never sees data beyond the classifier’s training set. Black tags mark the two factors we vary, and ⊕is the union of Eq. (1). The last box gives top-1 test accuracy.

Datasets

The study uses six datasets representing different challenges:

  • Benchmarks: CIFAR-10, CIFAR-100, MNIST, and Fashion-MNIST.
  • Medical Imaging: DermaMNIST (skin lesions) and BloodMNIST (blood cells), which feature class imbalances and higher stakes.

Key Insights and Ideas

  • The Room-to-Grow Hypothesis: GDA helps most when the "real-only" classifier is performing poorly. If a backbone is already highly accurate (like ResNet-18 on CIFAR-10), adding synthetic data is more likely to hurt than help.
  • Fidelity is Non-Negotiable: Low-fidelity generators like VAEs (which produce blurry images) almost universally degrade performance. High-fidelity models like Flow Matching and Diffusion are necessary but not sufficient for gains.
  • The "Traditional Augmentation" Benchmark: In many cases, simply using RandAugment (traditional transformations like rotation and cropping) achieves similar or better gains than GDA for a fraction of the computational cost.
  • Theoretical Trade-off: The paper's bound "charges" GDA for the discrepancy between real and generated distributions while "crediting" it for the increased sample size. If the discrepancy (Total Variation distance) is too large, the bound confirms that accuracy will drop.

Key Results

  • Mixed Success: As shown in Figure 1, accuracy moves in both directions. On CIFAR-100, high-fidelity GDA provided significant boosts (up to ~20 points for ConvNeXt-V2), but on MNIST and DermaMNIST, it often led to performance degradation.
  • Generator Ranking: Flow Matching and VP-SDE generally outperformed DDPM and VAE in terms of both sample quality (FID) and classifier downstream accuracy.
  • Distribution Shift Correlation: The authors found a strong correlation between the estimated lower bound of the distribution shift and the resulting accuracy gain. Generators that classifiers could easily distinguish from real data helped the least.
Figure 1. Accuracy moves in both directions as we add synthetic data. Test accuracy (%) against the mixing ratio α, at real-data fraction ρ = 1. Rows: backbones. Columns: datasets, each with its own vertical scale. Lines: generators, traditional augmentation and the real-only classifier (black), as three-seed means with ±1 standard-deviation bands. Values: Tabs. A1 to A3.
Generators with a larger estimated distribution discrepancy (further right on the x-axis) tend to provide lower accuracy gains.

Limitations and Future Work

  • Compute Budget: The authors fixed the number of training epochs. Since adding synthetic data increases the total number of images, GDA runs actually used more optimizer steps than real-only runs. Future work should compare these methods at a fixed step budget.
  • Measuring Discrepancy: The Total Variation distance is notoriously hard to calculate for high-dimensional image data. The authors used a lower-bound estimate from their own classifiers, but more precise metrics are needed.
  • Generator Optimization: The study used "off-the-shelf" generator configurations. It’s possible that tuning generators specifically for the downstream classification task (rather than just image fidelity) could yield better results.

Summary updated

Abstract

Generative data augmentation adds samples from generative models to a classifier's training set. Many influential studies use generators pre-trained on large external datasets, and so mix the value of synthetic data with knowledge transferred through pretraining. We study the case without external data, where each generator learns only from the classifier's own training set, and ask when and why adding generated samples to that training set raises or lowers the classifier's test accuracy. We train classifiers with four backbones on six datasets, two of them medical, adding samples from four generators. We bound the excess risk of a classifier trained with synthetic data, that is, how far its population risk can lie above the minimum risk within its hypothesis class. The bound charges synthetic data for the total variation distance between the real and generated distributions and credits synthetic data for a larger training set. The gain from adding synthetic data, the fall in population risk, is at most the excess risk of the real-only classifier, which trains on the real images alone. Read through the bound, the experiments suggest that synthetic data help most when our classifiers detect only a small discrepancy between the generated and real data and the real-only classifier has room to improve, and that synthetic data help little or hurt when the detected discrepancy is large. Generators with a larger lower bound on the total variation distance, a bound that we estimate from our own classifiers, tend to help less on five of the six datasets, and on the CIFAR datasets the most accurate backbone gains the least. The bound gives these directions but not the size of the largest gains, most of which traditional augmentation also reaches at the same number of optimizer steps.
BibTeX
@misc{kua2026generative,
  author = {Daniel Kua and Yan Song},
  title = {{Generative Data Augmentation Without External Data: An Empirical and Theoretical Study on Deep Networks}},
  year = {2026},
  eprint = {2611.21467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21467},
}
Figure