Exploring the Equivalence of Closed-Set Generative and Real Data Augmentation in Image Classification

Haowen Wang, Guowei Zhang, Xiang Zhang, Zeyuan Chen, Haiyang Xu, Dou Hoon Kwark, Zhuowen TuarXiv 2025 · DOI · arXiv

Abstract

This research investigates the efficacy of "closed-set" generative augmentation—the practice of training a generative model from scratch on a specific dataset to improve classification performance on that same dataset. By establishing a rigorous experimental framework across diverse domains including CIFAR-10 and BloodMNIST, the authors derive an empirical scaling law that quantifies the "exchange rate" between synthetic and real data. The study reveals that while synthetic and real data occupy separable manifolds, they share sufficient feature subspaces to provide a significant training signal. This approach proves particularly valuable for specialized datasets where large-scale pre-trained models often fail to produce relevant samples.

Introduction

In the current landscape of computer vision, "open-set" augmentation—leveraging massive models like Stable Diffusion pre-trained on internet-scale data—is a common practice. However, this approach often fails when applied to niche or specialized domains, such as medical imaging or specific industrial parts, where the pre-trained model lacks domain-specific knowledge.

This research addresses a fundamental question: Can a generative model learn "extra" information from a dataset that a standard classifier misses? If so, can we use that model to "hallucinate" useful new training examples from scratch? The authors move beyond qualitative assessments to provide a quantitative framework for comparing the utility of synthetic data against additional real data.

Methodology

The researchers established a controlled experimental environment to measure the impact of augmentation:

  1. Base Training: Models are initialized with a base training set ($n_{base}$).
  2. Generative Training: State-of-the-art generative models (such as EDM or DiT) are trained from scratch exclusively on the base set (closed-set).
  3. Augmentation Scenarios: Two expanded training sets are created—one augmented with additional real images ($n_{real+}$) and one with synthetic images ($n_{syn+}$).
  4. Classification Benchmarking: Standard classifiers (ResNet, ViT) are trained on these sets to compare accuracy gains.

The Equivalence Formula

A core contribution of this work is the derivation of an empirical scaling law that relates synthetic data requirements to real data gains:

$$\frac{n_{syn+}}{n_{base}} \simeq c_1 \tau^{n_{base}} \times \left(c_2^{\frac{n_{real+}}{n_{base}}} - 1\right)$$

This formula allows practitioners to calculate the "efficiency gap," determining exactly how much synthetic data is needed to match the performance boost of a specific amount of real data based on the size of the starting dataset.

Key Insights: Manifolds and Feature Subspaces

A critical finding of the study is the "Manifold Gap." Even when synthetic images appear high-quality, they live on a different mathematical manifold than real images. As illustrated in the figure below, a simple binary classifier can distinguish real from synthetic data with over 98% accuracy.

Analysis of data distributions: (a) shows that real and synthetic data occupy separable manifolds, while (b) demonstrates that they share similar feature subspaces, allowing for effective classification training.

Despite this separation (a), the data distributions share a significant "feature subspace" (b). This overlap ensures that the synthetic data provides a useful training signal for the target classification task.

Other key observations include:

  • Real Data Superiority: Real data is consistently more efficient; achieving the same accuracy boost always requires a larger volume of synthetic data.
  • The Scalability Paradox: Closed-set augmentation becomes more effective as the base training set grows. Larger base sets allow the generator to produce higher-quality, more diverse images that better capture the "tail" of the distribution.
  • Diminishing Returns: For a fixed base set, the marginal benefit of synthetic images decreases faster than that of real images, eventually hitting a performance ceiling.

Experimental Results

The study evaluated this approach across three primary datasets:

  • BloodMNIST: In this specialized medical domain, adding 4x synthetic data improved accuracy from 97.83% to 98.56%. Closed-set augmentation significantly outperformed open-set models, which often produced biologically inaccurate images.
  • CIFAR-10: Closed-set augmentation outperformed common open-set alternatives (like CIFAKE) because the locally-trained generator remained more faithful to the specific data distribution.
  • ImageNet-100: Tests on this mid-sized benchmark confirmed that the scaling laws hold as the complexity of the image domain increases.

Discussion and Limitations

While closed-set generative augmentation offers a powerful tool for data-starved specialized domains, it comes with significant computational costs. For instance, training the generator for ImageNet-100 required three days of computation on four A6000 GPUs. Furthermore, the current equivalence formulas are empirical, derived from curve fitting rather than fundamental mathematical principles of neural network learning. Future research is needed to determine how these scaling laws hold up at the "Foundation Model" scale involving billions of images. The authors suggest that improving the sample efficiency of generative models will be key to narrowing the efficiency gap between synthetic and real data.

Summary updated

Abstract

In this paper, we address a key scientific problem in machine learning: Given a training set for an image classification task, can we train a generative model on this dataset to enhance the classification performance? (i.e., closed-set generative data augmentation). We start by exploring the distinctions and similarities between real images and closed-set synthetic images generated by advanced generative models. Through extensive experiments, we offer systematic insights into the effective use of closed-set synthetic data for augmentation. Notably, we empirically determine the equivalent scale of synthetic images needed for augmentation. In addition, we also show quantitative equivalence between the real data augmentation and open-set generative augmentation (generative models trained using data beyond the given training set). While it aligns with the common intuition that real images are generally preferred, our empirical formulation also offers a guideline to quantify the increased scale of synthetic data augmentation required to achieve comparable image classification performance. Our results on natural and medical image datasets further illustrate how this effect varies with the baseline training set size and the amount of synthetic data incorporated.
BibTeX
@misc{wang2025exploring,
  author = {Haowen Wang and Guowei Zhang and Xiang Zhang and Zeyuan Chen and Haiyang Xu and Dou Hoon Kwark and Zhuowen Tu},
  title = {{Exploring the Equivalence of Closed-Set Generative and Real Data Augmentation in Image Classification}},
  year = {2025},
  eprint = {2508.09550},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.09550},
}
Figure