Scaling Visual Representations via Generative Data Diversity

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 128 papers40 conceptsfrozen 2026-10-04

For years, the progress of computer vision has been tethered to the scale of human-curated datasets. To build models like CLIP, researchers had to scrape billions of image-caption pairs from the internet. However, a new paradigm is emerging: instead of scouring the web for real data, we can use generative models to "imagine" synthetic training sets.

Recent work by Yonglong Tian, Lijie Fan, and colleagues suggests that synthetic data is not just a substitute for real data—it can be fundamentally better for representation learning. The key to this success lies in a counter-intuitive trade-off: to train a better vision model, you must prioritize visual diversity over image quality.

The Fidelity-Diversity Trade-off

When we use a diffusion model to generate images, we typically use a parameter called Classifier-Free Guidance (CFG). A high CFG scale forces the model to strictly follow the text prompt, producing "clean," high-fidelity images that look great to humans. A low CFG scale allows the model more creative freedom, resulting in noisier, more varied images that might not perfectly match the prompt.

In Fake it till you make it: Learning(s) from a synthetic ImageNet clone, Mert Bülent Sarıyıldız and Diane Larlus discovered that for training classifiers, high-fidelity images are actually detrimental. High guidance scales lead to "mode collapse," where the model generates repetitive, stereotypical versions of an object. By lowering the guidance scale, the generator produces a wider variety of visual representations. This diversity acts as a powerful form of natural data augmentation, forcing the downstream model to learn robust features that generalize to the messy variance of the real world.

StableRep: Multi-View Contrastive Learning

The most potent application of this idea is StableRep. Developed by researchers including Phillip Isola and Dilip Krishnan, StableRep treats the generative model as a "multi-view" engine.

In traditional contrastive learning, you take one real image and create two "views" by cropping or flipping it. In StableRep, you take one text prompt and generate multiple distinct images. Because these images all stem from the same prompt, they are semantically identical but visually diverse. Training a model to recognize that these different synthetic images represent the same concept allows it to learn high-level semantic representations more efficiently than it could from real-world data alone.

Scaling Laws and Robustness

Does synthetic data scale? In Scaling Laws of Synthetic Images for Model Training ... for Now, researchers found that synthetic data follows the same power-law performance trends as real data. While synthetic data currently requires more samples to match real-world supervised performance, it often surpasses real data in out-of-distribution (OOD) generalization.

Because generative models can be prompted to create rare or "long-tail" scenarios—like a "dog in a spacesuit" or specific medical conditions—they can fill gaps in real-world datasets. Joonghyuk Shin and Jaesik Park demonstrated this in Fill-Up, using synthetic data to balance long-tailed distributions and improve fairness and robustness in AI.

Beyond 2D Images

This "diversity-first" approach is spreading to other domains:

  • 3D Understanding: Kohei Torimi and Yuki Asano introduced TeGA, which uses text-to-3D models to augment point cloud datasets, achieving significant gains in zero-shot 3D classification.
  • Audio Generation: AudioTurbo by Junqi Zhao and Haohe Liu applies similar principles to text-to-audio, using low guidance scales to maintain creative diversity while accelerating generation speed.
  • Sampling Techniques: To avoid the fidelity-diversity trade-off entirely, Seyedmorteza Sadat proposed CADS, a sampling method that adds noise to the conditioning signal to boost diversity without sacrificing the sharp details provided by high guidance scales.

The Future of Model-to-Model Learning

The shift toward synthetic data suggests a future where the bottleneck for AI is no longer the availability of human-labeled images, but the diversity of generative prompts. As Zhuoran Yu argues in Diversify, Don't Fine-Tune, we don't necessarily need to fine-tune generative models on specific target domains. Instead, by using Large Language Models to generate a vast, diverse array of prompts, we can "unleash" the latent knowledge already stored in off-the-shelf diffusion models to train the next generation of visual learners.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_visual_representations_via_gener_763866ac,
  title = {Scaling Visual Representations via Generative Data Diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-visual-representations-via-generative-data-diversity/rdr_763866ac94c847feb9e5d7f5b2923e01},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_visual_representations_via_gener_763866ac_v1,
  title = {Scaling Visual Representations via Generative Data Diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-visual-representations-via-generative-data-diversity/rdr_763866ac94c847feb9e5d7f5b2923e01/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}