TL;DR
Jointly training continuous diffusion models alongside learnable continuous embeddings for categorical data often triggers embedding collapse, where the entire codebook collapses into a single point. This paper introduces CATDM, a continuous diffusion framework that pairs a novel Consistency-Matching regularizer with a shifted cosine noise schedule, simplified diffusion weighting, and random embedding dropout to stabilize training and enable progressive, high-fidelity discrete data generation.
What Problem Does This Paper Address?
Continuous-state diffusion models (CSDMs) excel at generating continuous data like images and audio, but adapting them to discrete or categorical data (e.g., text tokens, quantized image codes) remains tricky. A common workaround maps categorical tokens into a continuous embedding space where standard Gaussian diffusion can operate.
However, existing approaches typically freeze pretrained embeddings, forfeiting the performance gains of end-to-end learning. When researchers attempt to train the embedding codebook and the diffusion denoising network jointly, the optimization frequently suffers from embedding collapse: all codebook vectors converge to identical, constant representations. This trivializes the denoising task for the network, but destroys semantic structure and ruins generation quality.
Methods
The authors propose CATDM (Enhanced Embedding for CATegorical Data in Diffusion Models), incorporating four core components:
- Consistency-Matching (CM) Regularizer ($\mathcal{L}_{\text{CM}}$): To stabilize joint training, CATDM enforces that the model's categorical predictions along the Probability Flow ODE trajectory stay consistent across different timesteps $s < t$ using a KL-divergence loss with a stop-gradient EMA target. This propagates the supervision signal from the clean reconstruction loss at $t=0$ all the way to $t=1$.
- Rebalanced Loss Function ($\mathcal{L}_{\text{DM}}$): Instead of using the standard variational lower bound's signal-to-noise ratio weighting $-\text{SNR}'(t)$, which over-penalizes small noise levels and incentivizes collapse, CATDM uniformly weights the embedding prediction error across timesteps.
- Shifted Cosine Noise Schedule: By tuning a shifting parameter $s$ in $\log \text{SNR}(t) = -2\log\tan(\pi t / 2) + s$, the noise schedule allocates more capacity to high-noise regimes and injects noise faster near $t \approx 0$, preventing trivial denoising shortcuts.
- Random Dropping (RD): Masking a random subset of token embeddings during training with a special
[mask]token forces the codebook to organize semantically, placing tokens with similar contextual roles closer together in continuous space.
Datasets
The authors evaluate CATDM on three modalities using existing, standard benchmark datasets:
- Unconditional & Conditional Image Generation: FFHQ ($70\text{K}$ faces), LSUN Bedrooms ($3\text{M}$ images), LSUN Churches ($126\text{K}$ images), and ImageNet $256 \times 256$ ($1.2\text{M}$ images across $1{,}000$ classes), tokenized via a pretrained VQGAN ($16 \times 16$ discrete grid, codebook size $1{,}024$).
- Unconditional Text Generation:
text8, a character-level Wikipedia benchmark with a 27-token vocabulary (26 lowercase letters + whitespace). - Machine Translation: IWSLT14 German–English (De-En) and WMT14 English–German (En-De).
No new datasets were introduced.
Key Insights and Ideas
- Why collapse happens: The authors identify two culprits in standard variational diffusion: (1) the reconstruction loss only provides meaningful gradients near $t \approx 0$, failing to spread feedback across time; and (2) standard diffusion loss weighting drastically penalizes errors near $t \to 0$, tempting the optimizer to collapse embeddings into a zero-variance constant to quickly drive the loss down.
- Theoretical backing for CM: The authors prove (Theorem 1) that perfectly minimizing the consistency regularizer alongside zero-step reconstruction guarantees recovery of the exact data distribution ($P_{\phi^*, \theta^*} = P_{\text{data}}$).
- Connection to heuristic cross-entropy: Prior works used time-dependent cross-entropy losses as heuristic patches. The authors show (Proposition 2) that minimizing the CM regularizer theoretically subsumes this cross-entropy penalty while providing much stronger training dynamics in practice.
Key Results
- Image Generation: CATDM achieves state-of-the-art FID scores among discrete and continuous embedding baselines on unconditional tasks: 7.25 on FFHQ, 4.99 on LSUN Churches, and 4.16 on LSUN Bedrooms (surpassing D3PM, VQ-Diffusion, MaskGIT, and fixed-embedding CSDM). On class-conditional ImageNet $256 \times 256$, it achieves an FID of 6.81 and an Inception Score of 225.31 using only 50 sampling steps—matching or exceeding MaskGIT without relying on sampling heuristics like decaying Gumbel noise.
- Machine Translation: CATDM achieves BLEU scores of 29.61 (IWSLT14 De-En) and 21.67 (WMT14 En-De), outperforming all evaluated non-autoregressive and discrete/continuous diffusion baselines (including Difformer, CDCD, and DiffuSeq).
- Text Generation: On
text8, CATDM achieves a test BPC of $\le 1.54$, outperforming multinomial diffusion and uniform D3PM, though trailing autoregressive models.
Limitations and Future Work
- Autoregressive Gap in Text: Similar to other non-autoregressive models, CATDM lags behind standard autoregressive Transformers in raw language modeling perplexity (BPC) and translation quality, largely because non-autoregressive formulations struggle with duplicate or missing tokens.
- Architectural Scope: The empirical validation focused primarily on standard Transformer backbones; exploring alternative continuous architectures (e.g., U-Nets, state-space models) remains open.
- Future Modalities: The authors suggest extending CATDM to structured discrete domains such as graphs and raw audio codes, as well as developing advanced fast-sampling techniques tailored for embedding spaces.
Summary updated