Abstract
Recent text-to-image (T2I) diffusion models produce visually stunning images and demonstrate excellent prompt following. But do they perform well as synthetic vision data generators? In this work, we revisit the promise of synthetic data as a scalable substitute for real training sets and uncover a surprising performance regression. We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025, train standard classifiers solely on this synthetic data, and evaluate them on real test data. Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators. Our analysis reveals a hidden trend: These models collapse to a narrow, aesthetic-centric distribution that undermines diversity and real data distribution coverage. Overall, our findings challenge a growing assumption in vision research, namely that progress in generative realism implies progress in data realism. We thus highlight an urgent need to rethink the capabilities of modern T2I models as reliable training data generators.
Introduction
The "data hunger" of modern AI has led researchers to view synthetic data as the ultimate scalable solution. The prevailing assumption has been simple: as generative models get better at producing realistic images and following complex prompts, the synthetic datasets they produce should become better substitutes for real-world data. However, a significant disconnect is emerging. While modern models can generate a "photorealistic" goldfish with ease, these images often lack the specific features a classifier needs to recognize a real goldfish in the wild. This research investigates whether the rapid progress in T2I fidelity from 2022 to 2025 translates into better training utility, or if we are accidentally optimizing for "pretty" at the expense of "useful."
Methodology
The study conducted a large-scale benchmark of 13 state-of-the-art T2I models released between 2022 and 2025, including Stable Diffusion 1.5, SDXL, Flux.1-dev, and Qwen-Image. The experimental pipeline involved:
- Synthetic Dataset Generation: Creating large-scale datasets for a 200-class subset of ImageNet-1k using both simple class-name prompts and detailed LLM-generated captions.
- The "Synth → Real" Test: Training standard image classifiers (ResNet-50, ViT, ConvNeXt) solely on synthetic data and evaluating them on the real ImageNet-1k validation set.
- Diagnostic Probing: To identify the root causes of performance shifts, the authors analyzed the data across several dimensions: structure (depth maps), texture (local patches), and frequency domains (low-pass vs. high-pass filters).
Key Results: The Performance Regression
The most striking finding is a clear downward trend in "Synth → Real" accuracy over time. Models released in 2024 and 2025 consistently underperform older models like Stable Diffusion 1.5 when used as data generators with simple prompts. This suggests that as models are fine-tuned for human-centric aesthetic appeal (often via RLHF), they suffer from distribution collapse. They ignore the diverse, noisy variations found in nature in favor of a narrow, idealized style.
Furthermore, a "Prompting Paradox" exists: while newer models can produce better data when guided by extremely detailed LLM-generated captions, their performance on the simple prompts necessary for massive scaling is significantly worse than their predecessors.
Mechanisms of Failure: Texture vs. Structure
The diagnostic probing revealed that while newer models are excellent at capturing global structure (e.g., correct placement of eyes and limbs), they fail at high-frequency realism. Classifiers rely heavily on fine-grained textures to distinguish classes; when these textures become "too smooth" or artificial, the resulting model fails to generalize to real-world grit. This is evidenced by the fact that classifiers trained on synthetic structure (depth) show a much narrower gap to real data than those trained on texture.
Conclusion
This work highlights that sample realism does not equal data realism. An individual image from a model like Flux may look more "real" to a human, but as a collective dataset, it is often less diverse and informative for machine learning than the output of older, "cruder" models. The authors call for a shift in generative model evaluation, moving beyond photorealism toward "learnability-centered" metrics like density and coverage to ensure synthetic data remains a viable path for AI development.
Summary updated