Abstract
Diffusion models (DMs) have revolutionized generative learning by encoding data into a simple Gaussian distribution. However, mapping a complex, multimodal data distribution into a single continuous Gaussian prior represents an unnecessarily challenging learning problem. We propose Discrete-Continuous Latent Variable Diffusion Models (DisCo-Diff) to simplify this task by introducing complementary discrete latent variables. By augmenting DMs with learnable discrete latents inferred via an end-to-end trained encoder, we significantly reduce the geometric complexity (curvature) of the DM's generative ODE. DisCo-Diff does not rely on pre-trained networks, making it universally applicable across domains. Our results demonstrate state-of-the-art FID scores on class-conditioned ImageNet-64/128 and improved performance in specialized tasks like molecular docking.
Introduction
Standard Diffusion Models (DMs) operate by learning to reverse a diffusion process that gradually transforms data into Gaussian noise. While effective, this approach forces the model to learn a generative Ordinary Differential Equation (ODE) that maps a simple, unimodal distribution to a highly complex, multimodal one. This often results in high curvature in the generative trajectories, meaning the model must learn a "twisty" path from noise to data.
DisCo-Diff addresses this by introducing a hybrid latent structure. By partitioning the data space using a small set of discrete latent variables, the model can "straighten" these trajectories, making the generative process easier to learn and more efficient to sample.
The DisCo-Diff Framework
1. Hybrid Latent Structure
DisCo-Diff augments the standard continuous Gaussian noise with a small number of learnable discrete latents. While the continuous latents model local variability, the discrete latents are designed to capture global, multimodal structures—such as the general layout of an image or the specific binding mode of a molecule.
2. End-to-End Learning with Gumbel-Softmax
Unlike previous hybrid approaches that rely on fixed, pre-trained encoders (e.g., CLIP), DisCo-Diff trains its own encoder ($E_\phi$) jointly with the denoiser ($D_\theta$). To facilitate gradient flow through the discrete sampling step, the framework employs the Gumbel-Softmax relaxation. This allows the encoder to learn the most helpful discrete cues for the denoiser dynamically.
3. Two-Stage Training Pipeline
Training is conducted in two distinct phases:
- Stage 1 (Joint Optimization): The encoder and denoiser are co-optimized. The encoder learns to provide discrete codes that minimize the denoiser's reconstruction loss.
- Stage 2 (Prior Learning): A small, post-hoc autoregressive Transformer is trained to model the distribution of the learned discrete latents. This ensures that the discrete variables can be sampled effectively during inference.
4. Hierarchical Design
In its hierarchical variant, DisCo-Diff injects different groups of discrete latents into various resolutions of a U-Net architecture. This allows specific latents to specialize: some control high-level global layout and shape, while others handle fine-grained details like texture and color.
Experimental Results
Image Synthesis
DisCo-Diff achieves state-of-the-art results on competitive image benchmarks. On class-conditioned ImageNet-64, the model reached an FID of 1.65, and on ImageNet-128, it achieved 1.73 using an ODE sampler. Qualitative analysis shows that images sharing the same discrete latent often exhibit similar structural layouts or color palettes, even when they belong to different object classes, confirming that the discrete latents capture meaningful non-semantic variations.
Curvature Reduction
Using 2D toy data (Mixtures of Gaussians), the authors quantified the "straightness" of the generative paths. DisCo-Diff showed a massive reduction in the mean curvature of the generative ODE compared to standard DMs. Straighter paths are inherently easier for neural networks to approximate and allow for more accurate sampling with fewer steps.
Molecular Docking
To demonstrate universal applicability, the model was tested on the PDBBind dataset for molecular docking. DisCo-Diff improved the success rate (predicting ligand binding within 2Å of ground truth) from 32.9% to 35.4%. The gains were particularly pronounced on "unseen" proteins, suggesting that the discrete-continuous hybrid generalizes better to novel structures than purely continuous models.
Discussion: Why Discrete Latents?
A key finding of the research is the superiority of discrete auxiliary latents over continuous ones. Continuous auxiliary latents often suffer from "encoder collapse" (where the model ignores the latent) or extreme overfitting. Discrete latents, constrained by a codebook bottleneck, provide a more stable and sampleable prior for the second-stage Transformer.
Furthermore, because DisCo-Diff does not require pre-trained networks, it is uniquely suited for niche domains—like molecular biology or specialized medical imaging—where high-quality pre-trained encoders do not yet exist.
Conclusion
DisCo-Diff represents a significant step forward in making diffusion models more efficient and expressive. By simplifying the noise-to-data mapping through discrete partitions, the framework reduces the geometric complexity of the generative process. Future work will look toward a single-stage end-to-end training process and scaling the framework to large-scale text-to-image Transformer architectures.
Summary updated