Overview
Diffusion models (DMs) have revolutionized generative learning, but they face a fundamental challenge: mapping complex, multimodal data distributions into a single, unimodal continuous Gaussian distribution. DisCo-Diff (Discrete-Continuous Latent Variable Diffusion Models) introduces a hybrid approach to generative modeling by augmenting continuous diffusion models with learnable discrete latent variables. By allowing discrete latents to handle global, multimodal structures while the continuous process handles local details, the model significantly simplifies the learning task, resulting in straighter generative trajectories and state-of-the-art performance in both image synthesis and molecular docking.
The Challenge of Continuous Mapping
Standard DMs are incredibly powerful, but their reliance on a purely continuous process creates an unnecessarily challenging learning problem. Imagine trying to flatten a crumpled piece of paper into a perfect circle using only smooth, continuous movements. Because real-world data is highly complex, the mathematical path (the generative ODE) from noise back to data becomes highly "curved" and convoluted.
While we often use class labels or text prompts to guide generation, these semantic cues do not fully capture the structural variety within a category. For instance, as seen above, a "husky" can be running in the snow or lying down; these represent vastly different structural layouts despite sharing the same semantic label. Forcing a model to learn all these variations through a single continuous mapping requires many steps during sampling to maintain accuracy and makes training difficult.
The DisCo-Diff Approach
To address this, the authors propose splitting the "heavy lifting" of generation between discrete and continuous variables:
- Discrete Latents ($z$): A small set of learnable discrete variables captures global characteristics like layout, shape, and style.
- Continuous Latents: The standard diffusion process handles the local, continuous variations and fine details.
Key Technical Components: * End-to-End Training: Unlike previous methods that rely on frozen, pre-trained encoders (like CLIP), DisCo-Diff trains its encoder and denoiser together. This ensures the discrete latents learn exactly what the diffusion model needs to simplify its task. * Gumbel-Softmax Relaxation: To allow gradients to flow through the discrete "choices" made by the encoder during training, the authors utilize the Gumbel-Softmax trick. * Autoregressive Prior: Because the "clean" image is unavailable at inference time to run the encoder, a small Transformer is trained autoregressively to predict the distribution of these discrete latents. * Cross-Attention Injection: The discrete latents are fed into the denoiser neural network via cross-attention, guiding the continuous diffusion process.
Core Mechanisms and Insights
- Straightening the Path: The most striking insight is that discrete latents act as "anchors." By partitioning the data space into distinct modes, they significantly reduce the curvature of the generative ODE (as visualized in the architecture diagram). Straighter paths are easier for the model to learn and allow for faster, more stable sampling.
- Discrete vs. Continuous Structure: The authors argue that using continuous auxiliary latents merely shifts the burden—you would still need a complex model to learn the distribution of those latents. Discrete latents with small codebooks are much easier to model with a simple Transformer.
- Structural vs. Semantic: Discrete latents capture "structural" information that class labels miss. Even within the same class, different latents naturally cluster around different poses, layouts, or lighting conditions.
Empirical Performance
The researchers validated DisCo-Diff across three distinct domains, proving its versatility:
- Image Synthesis: DisCo-Diff achieved a new state-of-the-art (SOTA) FID score of 1.65 on class-conditioned ImageNet-64 and 1.73 on ImageNet-128 (using an ODE sampler). It also outperformed all baselines in unconditional settings, even those using pre-trained features.
- Molecular Docking: Demonstrating universality beyond computer vision, the model was tested on the PDBBind dataset (predicting how a ligand binds to a protein). The success rate (predictions within 2Å of ground truth) improved from 32.9% to 35.4% on the full test set, with an even larger jump for "unseen" proteins (13.9% to 18.5%).
- Efficiency: The extra overhead is minimal. Sampling the discrete latents via the Transformer takes a fraction of a second, while the main diffusion process still accounts for the bulk of the compute time.
Limitations and Future Directions
While highly effective, DisCo-Diff currently relies on a two-stage training pipeline (training the DM/Encoder first, followed by the Autoregressive prior). Moving toward a seamless, single-stage end-to-end training process is a natural next step. Additionally, on ImageNet-128, the model performed slightly worse with stochastic (DDPM) samplers compared to ODE samplers, suggesting the model might over-emphasize global structural information at the expense of high-frequency details in certain stochastic settings. Future work could also explore scaling this framework to massive text-to-image models or other modalities like speech and 3D video, where multimodal distributions are even more prevalent.
Summary updated