Controllable Generative Modeling without Human Labels

This direction is split into other directions: label-dependent synthesis. Its last version stays readable and citable here.

Version 1 of 1142 papers142 conceptsfrozen 2026-10-04

The primary bottleneck in modern generative AI is no longer the architecture, but the label. While models like Stable Diffusion or GPT-4 can generate nearly anything, steering them toward specific, high-fidelity outcomes typically requires massive datasets of paired images and text, or dense semantic masks. In specialized domains like medical imaging, game design, or robotics, these labels are either prohibitively expensive or non-existent.

A growing body of research is focused on breaking this "label bottleneck" by replacing human-annotated supervision with self-supervised representations, synthetic labeling pipelines, and reward-guided exploration. The goal is to achieve the precision of conditional models using the vast, untapped oceans of unlabeled data.

The Circular Dependency of Synthetic Data

To train a model that generates realistic medical images, you often need a dataset of real medical images with expert labels (like tumor segmentations). If you had those labels, you might not need the synthetic data in the first place. This circular dependency is a recurring theme in recent work.

For example, Hadrien Reynaud and colleagues note in Label-free Motion-Conditioned Diffusion Model for Cardiac Ultrasound Synthesis that the very labels scarce in the real world are usually required to generate the synthetic data intended to alleviate that scarcity.

To break this loop, researchers are moving toward self-labeling and mutual learning. In the Dual Pseudo Training (DPT) framework, Fan Bao and others demonstrate a "win-win" cycle: a classifier generates pseudo-labels to train a diffusion model, which in turn generates more synthetic data to improve the classifier. This allows models to reach high performance with as little as one human label per class.

Representations as Proxies for Labels

If human labels are unavailable, the model can instead look at the "geometry" of the data itself. Self-supervised learning (SSL) allows models to learn the underlying structure of a dataset—distinguishing a "cat" from a "dog" based on visual features rather than a text tag.

Tianhong Li and the team at MIT CSAIL pioneered this with Representation-Conditioned Generation (RCG). Instead of conditioning a model on a class label like "Golden Retriever," they condition it on a self-supervised embedding. This "proxy" label allows unconditional models to finally rival the quality of labeled ones, improving ImageNet generation quality by over 60% without using a single human-provided category.

Similarly, in the realm of image restoration, Junwei Zhu introduced SPL, which distills "semantic priors" from pre-trained classification networks. By injecting these learned priors into the generation process, the model can reconstruct complex scenes with structural coherence, bypassing the need for the dense edge maps or segmentation masks required by previous methods.

Steering via Rewards and Geometry

Another way to bypass labels is to define what a "good" output looks like through a reward function or a mathematical property, rather than a category name.

Reward-Guided Exploration

In Towards Controllable Diffusion Models via Reward-Guided Exploration, Hengtong Zhang and Tingyang Xu bridge generative modeling with Reinforcement Learning (RL). Instead of telling the model "generate a molecule with this label," they give the model a reward for generating molecules with specific physical properties. This allows the model to optimize for complex, non-differentiable goals that were never explicitly labeled in the training set.

Training-Free Guidance

Sometimes, the "labels" are already hidden inside the model's own weights. Eric Yeats discovered that diffusion models often fail because they get "stuck" in low-probability regions of their own internal data manifold. His Saddle-Free Guidance (SFG) is a training-free method that uses the model's own log-density gradients to steer samples toward high-fidelity regions. This improves image quality and diversity without requiring any external classifiers or labeled datasets.

Scaling to Specialized Domains

The impact of label-free control is most visible in fields where data is "locked" or expensive.

  • Game Design: Yuhe Nie and the "Moonshine" team distilled procedural algorithms into steerable models by using LLMs to synthetically label thousands of procedurally generated maps, allowing designers to use natural language prompts for game world generation.
  • Medical Video: Hadrien Reynaud developed EchoFlow, which generates synthetic echocardiogram videos. By using latent flow matching instead of manual masks, the system produces synthetic data so realistic that AI models trained on it perform just as well as those trained on real, private patient data.
  • Computer Vision: Austin Xu and Mariya I. Vasileva created HandsOff, a framework that uses GAN inversion to generate infinite labeled datasets for tasks like depth estimation, requiring fewer than 50 initial human annotations.

By shifting the focus from collecting labels to inferring or synthesizing them, this research direction enables high-fidelity, controllable AI to move beyond general-purpose image generation and into the complex, label-scarce realities of science and industry.

Papers in this version

Showing 60 of 142 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_controllable_generative_modeling_without_d79439c1,
  title = {Controllable Generative Modeling without Human Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-generative-modeling-without-human-labels/rdr_d79439c1400e4e89853e44c5025dea9c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_controllable_generative_modeling_without_d79439c1_v1,
  title = {Controllable Generative Modeling without Human Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-generative-modeling-without-human-labels/rdr_d79439c1400e4e89853e44c5025dea9c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}