DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents

Yilun Xu, Gabriele Corso, Tommi S. Jaakkola, Arash Vahdat, Karsten Kreis

Abstract

Diffusion models (DMs) have revolutionized generative learning by encoding data into a simple Gaussian distribution. However, mapping a complex, multimodal data distribution into a single continuous Gaussian prior represents an unnecessarily challenging learning problem. We propose Discrete-Continuous Latent Variable Diffusion Models (DisCo-Diff) to simplify this task by introducing complementary discrete latent variables. By augmenting DMs with learnable discrete latents inferred via an end-to-end trained encoder, we significantly reduce the geometric complexity (curvature) of the DM's generative ODE. DisCo-Diff does not rely on pre-trained networks, making it universally applicable across domains. Our results demonstrate state-of-the-art FID scores on class-conditioned ImageNet-64/128 and improved performance in specialized tasks like molecular docking.

Introduction

Standard Diffusion Models (DMs) operate by learning to reverse a diffusion process that gradually transforms data into Gaussian noise. While effective, this approach forces the model to learn a generative Ordinary Differential Equation (ODE) that maps a simple, unimodal distribution to a highly complex, multimodal one. This often results in high curvature in the generative trajectories, meaning the model must learn a "twisty" path from noise to data.

DisCo-Diff addresses this by introducing a hybrid latent structure. By partitioning the data space using a small set of discrete latent variables, the model can "straighten" these trajectories, making the generative process easier to learn and more efficient to sample.

The DisCo-Diff Framework

1. Hybrid Latent Structure

DisCo-Diff augments the standard continuous Gaussian noise with a small number of learnable discrete latents. While the continuous latents model local variability, the discrete latents are designed to capture global, multimodal structures—such as the general layout of an image or the specific binding mode of a molecule.

2. End-to-End Learning with Gumbel-Softmax

Unlike previous hybrid approaches that rely on fixed, pre-trained encoders (e.g., CLIP), DisCo-Diff trains its own encoder ($E_\phi$) jointly with the denoiser ($D_\theta$). To facilitate gradient flow through the discrete sampling step, the framework employs the Gumbel-Softmax relaxation. This allows the encoder to learn the most helpful discrete cues for the denoiser dynamically.

3. Two-Stage Training Pipeline

Training is conducted in two distinct phases:

  • Stage 1 (Joint Optimization): The encoder and denoiser are co-optimized. The encoder learns to provide discrete codes that minimize the denoiser's reconstruction loss.
  • Stage 2 (Prior Learning): A small, post-hoc autoregressive Transformer is trained to model the distribution of the learned discrete latents. This ensures that the discrete variables can be sampled effectively during inference.

4. Hierarchical Design

In its hierarchical variant, DisCo-Diff injects different groups of discrete latents into various resolutions of a U-Net architecture. This allows specific latents to specialize: some control high-level global layout and shape, while others handle fine-grained details like texture and color.

Experimental Results

Image Synthesis

DisCo-Diff achieves state-of-the-art results on competitive image benchmarks. On class-conditioned ImageNet-64, the model reached an FID of 1.65, and on ImageNet-128, it achieved 1.73 using an ODE sampler. Qualitative analysis shows that images sharing the same discrete latent often exhibit similar structural layouts or color palettes, even when they belong to different object classes, confirming that the discrete latents capture meaningful non-semantic variations.

Curvature Reduction

Using 2D toy data (Mixtures of Gaussians), the authors quantified the "straightness" of the generative paths. DisCo-Diff showed a massive reduction in the mean curvature of the generative ODE compared to standard DMs. Straighter paths are inherently easier for neural networks to approximate and allow for more accurate sampling with fewer steps.

Molecular Docking

To demonstrate universal applicability, the model was tested on the PDBBind dataset for molecular docking. DisCo-Diff improved the success rate (predicting ligand binding within 2Å of ground truth) from 32.9% to 35.4%. The gains were particularly pronounced on "unseen" proteins, suggesting that the discrete-continuous hybrid generalizes better to novel structures than purely continuous models.

Discussion: Why Discrete Latents?

A key finding of the research is the superiority of discrete auxiliary latents over continuous ones. Continuous auxiliary latents often suffer from "encoder collapse" (where the model ignores the latent) or extreme overfitting. Discrete latents, constrained by a codebook bottleneck, provide a more stable and sampleable prior for the second-stage Transformer.

Furthermore, because DisCo-Diff does not require pre-trained networks, it is uniquely suited for niche domains—like molecular biology or specialized medical imaging—where high-quality pre-trained encoders do not yet exist.

Conclusion

DisCo-Diff represents a significant step forward in making diffusion models more efficient and expressive. By simplifying the noise-to-data mapping through discrete partitions, the framework reduces the geometric complexity of the generative process. Future work will look toward a single-stage end-to-end training process and scaling the framework to large-scale text-to-image Transformer architectures.

Summary updated

Abstract

Diffusion models (DMs) have revolutionized generative learning. They utilize a diffusion process to encode data into a simple Gaussian distribution. However, encoding a complex, potentially multimodal data distribution into a single continuous Gaussian distribution arguably represents an unnecessarily challenging learning problem. We propose Discrete-Continuous Latent Variable Diffusion Models (DisCo-Diff) to simplify this task by introducing complementary discrete latent variables. We augment DMs with learnable discrete latents, inferred with an encoder, and train DM and encoder end-to-end. DisCo-Diff does not rely on pre-trained networks, making the framework universally applicable. The discrete latents significantly simplify learning the DM's complex noise-to-data mapping by reducing the curvature of the DM's generative ODE. An additional autoregressive transformer models the distribution of the discrete latents, a simple step because DisCo-Diff requires only few discrete variables with small codebooks. We validate DisCo-Diff on toy data, several image synthesis tasks as well as molecular docking, and find that introducing discrete latents consistently improves model performance. For example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned ImageNet-64/128 datasets with ODE sampler.
BibTeX
@inproceedings{xu2024disco,
  author = {Yilun Xu and Gabriele Corso and Tommi S. Jaakkola and Arash Vahdat and Karsten Kreis},
  title = {{DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=psup68MBvt},
}
Figure