Overcoming the Annotation Bottleneck in Controllable Generation

68 concepts71 papers1 proposal

Conditional generative models allow us to steer data synthesis by providing explicit guidance, such as a class label, a text prompt, or a spatial layout. However, training these models traditionally introduces a demanding prerequisite: massive, high-fidelity, manually annotated datasets.

To map a semantic concept to a synthetic output, a model must observe thousands of paired examples. When fine-grained control is required—such as specifying the exact bounding box of an object, the continuous density of a medical anomaly, or a niche text attribute—the required annotations become dense, expensive, and often unavailable. This reliance on human supervision creates a severe annotation bottleneck, limiting the scalability of generative models across specialized domains.

To break this dependency, a prominent research direction focuses on synthesizing data with minimal, noisy, or entirely absent human labels. Rather than relying on external human annotation, these approaches discover semantic structure directly from the data's underlying geometry, distill spatial-semantic priors from pre-trained networks, or maximize the utility of sparse supervision.


Discovering Structure from Within: Self-Conditioning and Latent Steering

The core intuition behind unsupervised controllable generation is that unlabeled data already possesses an implicit geometric structure. An unconditional model must internalize this structure to generate realistic samples. Therefore, we can extract and repurpose this internal knowledge as a conditioning signal without ever introducing human labels.

Clustering Feature Spaces

One way to generate distinct categories without labels is to let the model discover its own "pseudo-classes." During unconditional training, a discriminator learns to differentiate real images from fake ones by extracting highly discriminative features.

Liu et al. leverage this by iteratively clustering real data within the discriminator's feature space. By treating these clusters as pseudo-labels, they train the generator to target specific clusters, significantly improving image diversity and mitigating mode collapse without manual annotation. Similarly, Chen et al. integrate K-Means clustering directly into the generative adversarial network (GAN) framework, demonstrating that unsupervised cluster assignments can guide image synthesis as effectively as explicit class labels.

Unlabeled Data ──> Discriminator Feature Space ──> K-Means Clustering ──> Pseudo-Classes ──> Conditional Generator

For continuous control (such as adjusting the pose of a face or the size of a molecule), researchers look to the latent space of pre-trained models. Deep generative models tend to organize variations smoothly along continuous trajectories.

Zheng et al. show that by combining unsupervised 3D reconstructions with a decorrelation loss, they can isolate orthogonal editing directions in a GAN's latent space—such as lighting or head rotation—without a single manual label. This concept extends beyond images: Liu et al. introduce a model-agnostic framework that discovers "steerable factors" in graph deep generative models, allowing researchers to precisely edit molecular properties without expert-labeled datasets.


Borrowing Knowledge: Distillation and Multimodal Priors

When we lack labeled data for a specific target task, we do not have to start from scratch. Instead, we can transfer or distill the rich, structured representations embedded within existing foundational models.

Exploiting Cross-Attention Maps

Text-to-image diffusion models learn a powerful implicit alignment between words and pixels. To generate an image of a "cat," the model must internally track which pixels correspond to the word "cat."

Wu et al. exploit this mechanism to bypass manual pixel-level segmentation. By extracting the internal cross-attention maps of a pre-trained diffusion model during generation, they automatically synthesize high-quality images paired with precise, pixel-level semantic masks. This eliminates the need for human-labeled seed images to train mask decoders, creating an autonomous pipeline for generating labeled training data.

Text Prompt ──> Diffusion Model ──> [ Internal Cross-Attention Layers ] ──> Extracted Pixel Masks
                                   └──> Generated Synthetic Image ───────┘

Distilling Unconditional Models into Conditional Ones

Unconditional models are often easier to train on massive datasets because they require no labels, but they lack user control. To bridge this gap, the authors of Distilling GANs with Style-Mixed Triplets for X2I Translation with Limited Data propose a method to distill a high-quality unconditional model (like StyleGAN) into a conditional image-to-image translation network. By generating an infinite supply of synthetic, "style-mixed" image triplets, they provide the conditional model with a dense training signal, achieving strong performance even when real paired data is extremely limited.

Leveraging Multimodal Embeddings

Vision-language models like CLIP map images and text into a shared semantic space. This shared space can act as an unsupervised supervisor. Liu et al. introduce Semantic Diffusion Guidance, which uses a self-supervised, noised version of CLIP to steer diffusion models using text or image prompts without requiring text annotations on the target dataset.

Similarly, Abdal et al. link StyleGAN's latent space with CLIP to automatically discover, isolate, and name semantic editing directions entirely from scratch, removing the human from the loop of defining and labeling attributes.


Maximizing Scarce Supervision: Few-Shot and Semi-Supervised Guidance

In many practical scenarios, labels are not entirely absent, but they are highly scarce, noisy, or expensive to obtain. A key focus of this research direction is developing architectures that maximize the leverage of every single available label.

Semi-Supervised and Self-Supervised Objectives

Rather than forcing a discriminator to only evaluate whether an image is "real and matching a label," models can be augmented with self-supervised auxiliary tasks. The authors of Self-Supervised GANs via Auxiliary Rotation Loss introduce a task where the discriminator must predict the rotation angle of an image. This simple self-supervised signal stabilizes training and bridges the performance gap between unconditional and conditional GANs without requiring manual data annotation.

When high-fidelity generation is required, Lucic et al. demonstrate that combining self-supervised rotation losses with semi-supervised learning allows models to match the performance of fully supervised state-of-the-art architectures while using only 10% to 20% of the labeled data.

Learning from Noisy and Uncurated Data

Real-world datasets are frequently plagued by mislabeled samples or irrelevant background data. To prevent these imperfections from degrading conditional synthesis, Katsumata proposes a soft curriculum framework. By using an auxiliary classifier to dynamically weight and correct training instances, the model learns effectively from noisy labels and uncurated data, achieving high-quality conditional generation with a fraction of the clean expert annotations typically required.

Few-Shot Latent Priors

When only a handful of labeled examples exist (e.g., fewer than 100), standard conditional guidance techniques fail due to overfitting. To combat this, Sinha et al. introduce D2C, a framework that trains a diffusion-based prior over a robust, contrastive latent space. Because the latent space is already organized by self-supervised contrastive learning, the model can achieve fine-grained attribute manipulation and conditional generation using as few as 100 labeled examples.

For post-hoc steering of diffusion models, Zhao et al. frame conditional control as a reinforcement learning problem. Their framework, CTRL, bypasses complex fine-tuning by using small offline datasets to efficiently steer pre-trained models, demonstrating that precise attribute control can be achieved as long as the target attribute is conditionally independent of the base prompt.

Key Foundational Work

Learning to Generate Chairs, Tables and Cars with Convolutional Networks (2017) — contributed 1 concept elements to this direction.
Structured Generative Adversarial Networks (2017) — contributed 1 concept elements to this direction.
Guiding InfoGAN with Semi-supervision (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_label_dependent_synthesis_d6207637,
  title = {label-dependent synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-dependent-synthesis/rdr_d6207637582645668a14d72b787446ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_label_dependent_synthesis_d6207637_v1,
  title = {label-dependent synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-dependent-synthesis/rdr_d6207637582645668a14d72b787446ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake