Contrastive Conditional–Unconditional Alignment for Long-tailed Diffusion Model

Fang Chen, Alex Villa, Gongbo Liang, Xiaoyi Lu, Meng TangSubmission to ICLR 2026 · Rejected

TL;DR

Researchers have developed a new framework called CCUA (Contrastive Conditional-Unconditional Alignment) to fix the "mode collapse" problem in diffusion models trained on imbalanced data. By combining a contrastive loss that forces diversity with an alignment loss that lets rare classes "borrow" structural knowledge from common ones, they significantly improved the quality and variety of images generated for underrepresented categories.

What Problem Does This Paper Address?

In the real world, data is rarely balanced. Most datasets follow a "long-tailed" distribution: a few "head" classes have thousands of examples (like dogs or cats), while many "tail" classes have only a handful (like specific rare bird species or niche medical conditions).

When you train a class-conditional diffusion model (like Stable Diffusion or DiT) on this imbalanced data, it struggles with the tail. Because it has seen so few examples of the rare classes, it often suffers from mode collapse—generating the exact same image over and over—or producing low-quality, noisy results. Previous attempts to fix this often involved complex data augmentation or compromised the quality of the common "head" classes.

Methods

The authors introduce two elegant loss functions that can be plugged into existing architectures like U-Nets or Diffusion Transformers (DiT):

  1. Unsupervised Contrastive Loss (UCL): This acts as a "repulsive force." It looks at the latent representations of images within a training batch and pushes them away from each other. Crucially, it is unsupervised, meaning it encourages diversity across the board without getting bogged down by class labels that might be scarce for tail categories.
  2. Alignment Loss (AL): This is the "knowledge sharing" component. In the early stages of the diffusion process (when the image is mostly noise), the model doesn't really need to know the specific class to get the basic shapes right. The AL forces the model to align its "class-specific" generation with its "general" (unconditional) generation during these initial steps. This allows rare classes to benefit from the rich structural information the model learned from the abundant head classes.

These are combined with Batch Resampling, a standard trick where the model is shown tail-class images more frequently during training to ensure it doesn't ignore them.

Datasets

The researchers evaluated their method on several "Long-Tailed" (LT) versions of popular benchmarks:

  • ImageNet-LT (256×256): The primary high-resolution benchmark.
  • TinyImageNet-LT & Places-LT: Used to test performance on different scene types and smaller scales.
  • CIFAR10-LT & CIFAR100-LT: Standard small-scale datasets for rapid iteration and statistical analysis.

Key Insights and Ideas

  • The Early-Stage Similarity: The authors observed that at high noise levels (large timesteps), the low-frequency components of images (coarse shapes and backgrounds) are remarkably similar across all classes. By forcing alignment here, the model learns a "universal" foundation that supports rare classes.
  • Avoiding the Contrastive Shortcut: Standard supervised contrastive learning often takes a "shortcut" by just pushing different classes apart while letting images within a class stay identical. By applying the contrastive loss to the unconditional latent space and then "distilling" that diversity into the conditional space via the Alignment Loss, the authors forced the model to achieve true intra-class diversity.
  • Architecture Agnostic: The method works equally well on older U-Net-based models (DDPM) and the newer, state-of-the-art Diffusion Transformers (DiT/SiT).

Key Results

The CCUA framework showed dramatic improvements across the board, particularly in Recall (a measure of diversity) and FID (a measure of image quality).

  • ImageNet-LT (256x256): Compared to the baseline (SiT), CCUA improved the tail-class FID from 33.9 down to 19.5—a massive leap in quality for rare categories.
  • Diversity Boost: On the same dataset, the model achieved a 10% absolute improvement in Recall, meaning it generated a much wider variety of images for rare classes rather than repeating the same few patterns.
  • Visual Fidelity: Qualitative results show that while standard models might generate a blurry blob for a rare class like "red wine," CCUA generates distinct, high-quality images of glasses, bottles, and varying lighting conditions.

Limitations and Future Work

The primary drawback is computational overhead. Because the method requires the model to perform both a conditional and an unconditional pass during training to calculate the alignment, it increases training time. The authors noted that their original implementation was about 1.6x slower than standard training, though they managed to optimize this down to 1.3x slower using a batch-concatenation trick.

Future work could involve exploring more efficient ways to achieve this alignment or applying these "long-tail" fixes to even larger-scale foundational models where data imbalance remains a persistent challenge for fairness and representation.

Summary updated

Abstract

Training data for class-conditional image synthesis often exhibit a long-tailed distribution with limited images for tail classes. Such an imbalance causes mode collapse and reduces the diversity of synthesized images for tail classes. For class-conditional diffusion models trained on imbalanced data, we aim to improve the diversity and fidelity of tail class images without compromising the quality of head class images. We achieve this by introducing two simple but highly effective loss functions. Firstly, we employ an Unsupervised Contrastive Loss (UCL) utilizing negative samples to increase the distance/dissimilarity among synthetic images. Such regularization is coupled with a standard trick of batch resampling to further diversify tail-class images. Our second loss is an Alignment Loss (AL) that aligns class-conditional generation with unconditional generation at large timesteps. This second loss makes the denoising process insensitive to class conditions for the initial steps, which enriches tail classes through knowledge sharing from head classes. We successfully leverage contrastive learning and conditional-unconditional alignment for class-imbalanced diffusion models. Our framework is easy to implement as demonstrated on both U-Net based architecture and Diffusion Transformer. Our method outperforms vanilla denoising diffusion probabilistic models, score-based diffusion model, and alternative methods for class-imbalanced image generation across various datasets, in particular ImageNet-LT with 256×256 resolution.
BibTeX
@misc{chen2025contrastive,
  author = {Fang Chen and Alex Villa and Gongbo Liang and Xiaoyi Lu and Meng Tang},
  title = {{Contrastive Conditional–Unconditional Alignment for Long-tailed Diffusion Model}},
  year = {2025},
  eprint = {2507.09052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09052},
}

Loading related content...

Figure