Cluster Labels as a Diversity Prior for Few-Step Distilled Generators

Fast image generators are usually made by distillation. A slow "teacher" diffusion model is compressed into a "student" that produces an image in one to four steps. A diffusion model is a generator that turns random noise into an image by removing noise a little at a time. Students made this way lose much of their variety: ask for many samples and they keep producing near-copies of the same layout. Distilling Diversity and Control in Diffusion Models traced this to a single moment. Distilled students commit to the global structure of the image at their very first step, while the teacher spreads that decision over many steps. Their fix runs the full teacher for the first step and hands off to the student. This restores diversity, but it requires both models in GPU memory at once.

We propose to move that first structural decision out of the network and into a label. The label is not a human annotation. It is a cluster index found without supervision, by grouping self-supervised image features. The student is conditioned on this index, and at sampling time the index is drawn from the empirical cluster frequencies before the student runs. A categorical draw costs nothing at inference, and if the clusters capture coarse structure, the draw can stand in for the teacher's first step. The central claim is a tradeoff we expect to find and measure. Finer clusters leave less variety inside each condition for the student to collapse, which closes the student–teacher diversity gap. But finer clusters also push samples away from the real data, an effect documented by Rethinking Cluster-Conditioned Diffusion Models for Label-Free Image Synthesis. We predict that the best cluster count for a few-step student is larger than the best count for its multi-step teacher.

Core Claim

Few-step students lose diversity because one network evaluation cannot represent a target that has many distinct modes; it settles on a few. Conditioning on a label lowers the number of modes per condition. A label-free cluster index is therefore a cheap way to recover diversity lost in distillation. The benefit grows with cluster granularity until the distribution shift caused by fine clustering dominates. The claim is falsified if any of three things happens. A cluster-conditioned student may show no smaller diversity gap than an unconditional one. The gap may shrink just as much with random partitions as with semantic clusters, meaning the effect is plain conditioning noise rather than structure. Or, at equal memory, the gain may be fully explained by the student simply having a conditioning input.

The ingredients come from prior work. Cluster-conditioned diffusion (C-EDM) clusters DINO ViT-B features with TEMI, a self-supervised clustering method that forces clusters to be used evenly. C-EDM matches or beats ground-truth labels for multi-step models, reaching FID 1.67 on CIFAR-10 and 2.17 on CIFAR-100, where FID (Fréchet Inception Distance) measures how far generated images are from real ones, lower being better. It also warns that very fine clusters raise out-of-distribution scores. High-Fidelity Image Generation With Fewer Labels shows that hard pseudo-labels from clustering can replace most human labels in GANs. C-EDM motivates the proposal, and its granularity warning constrains it. Neither work studies few-step students, where we expect the granularity tradeoff to shift.

Conditional Diffusion Distillation (CoDi) supplies the distillation method. It trains a few-step conditional student with a consistency loss, which teaches the student to map any noisy point on a sampling path straight to the clean endpoint. It does this with a frozen backbone, a lightweight condition adapter and a latent $\ell_2$ loss, reaching FID 19.637 in four steps on DF2K super-resolution against a 250-step teacher's 19.200. CoDi assumes paired human conditions such as depth maps. We swap those for discovered cluster indices, and we target diversity rather than speed alone.

The diversity distillation paper supplies both the diagnosis and the main competitor. Our approach differs in keeping a single model in memory.

Research Program

The pipeline has three stages, all label-free. First, cluster the training images with TEMI on DINO features at a sweep of granularities $C$. For each data set, include the ground-truth class count and multiples above it.

Second, train a cluster-conditioned multi-step teacher per granularity following the C-EDM recipe. This stage is cheap at the image sizes we target.

Third, distill each teacher into one-, two- and four-step students with a conditional consistency loss in the style of CoDi, with the cluster index fed through the same embedding the teacher uses. At sampling time the student draws $c \sim \hat p(c)$, the empirical cluster frequencies, then draws noise and generates.

The analysis rests on one quantity: the diversity gap $\Delta(C)$. It is the teacher's average pairwise perceptual distance between samples minus the student's. Perceptual distance is measured with DreamSim, a learned image-similarity metric. We measure the gap both within clusters and across the full sample. We plot $\Delta(C)$ against the out-of-distribution AUROC used by C-EDM, at every step count.

We also reuse the diversity paper's $\phi_0$ visualization. It shows the model's estimate of the clean image at each intermediate noise level. We use it to check whether, at fixed $c$, student first-step estimates still vary across noise draws, or whether all variety now comes from $c$.

We will also test a soft-label variant. The student is conditioned on the cluster posterior, the probability of each cluster given the image, instead of a single index. This addresses an open conflict in the literature. Classic distillation holds that soft targets carry useful "dark knowledge" about which classes resemble each other, as argued in Transferring knowledge from a RNN to a DNN. S³GAN reports the opposite for generative conditioning: hard labels beat soft ones. If soft labels blur modes back together, they should widen $\Delta$ and so lose. We will run this test once.

Baselines

The naive reference point is an unconditional teacher distilled into an unconditional few-step student. It uses the same architecture, the same consistency loss and the same number of training images. Every conditioned variant must shrink this model's diversity gap without raising FID to count as an improvement.

The strongest existing method for restoring student diversity is the hybrid sampler from Distilling Diversity and Control in Diffusion Models. It runs the multi-step teacher for the first step and the student for the rest. We apply it to our unconditional teacher and student. We report it twice: at its native memory cost of two models, and at matched memory, where our student is widened to the combined parameter count. The paper reports restored DreamSim diversity on COCO-10k and comparable FID on COCO-30k, but the file gives no numeric values for either: no reported result found on Lacuna for that metric. We will therefore establish its numbers ourselves. Beating it means matching its diversity within statistical error while holding one model, or exceeding it at matched memory.

The oracle is a teacher and student conditioned on ground-truth CIFAR classes. The quality ceiling for the multi-step teacher is C-EDM's reported FID of 1.67 on CIFAR-10 and 2.17 on CIFAR-100. Few-step results on these data sets: no reported result found on Lacuna.

The key ablation conditions the student on a random partition of the data with the same $C$ and the same cluster sizes. This isolates whether semantic structure, rather than an extra input, does the work. A second ablation compares hard and soft cluster conditioning.

All variants share the same data, training steps, tuning budget for the learning rate and loss weights, and sampling steps.

Why not just…?

Why not just use the hybrid teacher-first-step sampler? It works, and it is a real threat on diversity alone. Its cost is keeping both networks in memory, which Distilling Diversity and Control in Diffusion Models itself names as the obstacle to edge deployment. The matched-memory comparison settles this. If a widened single student with clusters cannot match hybrid diversity, we will report that the label carries less structure than one teacher step.

Why not just consistency-distill an existing cluster-conditioned model? That is essentially our pipeline. We claim no new architecture. What is new is the prediction that the best granularity shifts upward under distillation, along with the measured tradeoff between diversity gap and distribution shift. If $\Delta(C)$ turns out to be flat in $C$, that prediction fails and the result reduces to a recipe.

Why not just add a diversity loss or use more steps? Distilling GANs with Style-Mixed Triplets for X2I Translation with Limited Data uses a loss that forces outputs to differ when the conditioning input changes. That pushes outputs apart without regard to whether they stay realistic. We include it as an added baseline under the same tuning budget and compare FID at equal diversity. More steps is the trivial fix; we report the full diversity-versus-steps curve so the reader can see at what step count clusters stop mattering.

Proposed Experiments

We will run on CIFAR-10, CIFAR-100 and FFHQ at 64×64, the settings where C-EDM established cluster conditioning. Metrics are FID, DreamSim pairwise distance within and across clusters, and C-EDM's out-of-distribution AUROC.

Two checks are decisive. First, at one and two steps, the semantic-cluster student must show a smaller diversity gap than both the unconditional student and the random-partition student, at equal or better FID. Second, the $C$ that minimizes FID for the student must be larger than the teacher's best $C$. Failing the first check refutes the mechanism. Failing the second means distillation does not change the granularity tradeoff, and the method is only a convenience.