Mitigating Embedding Collapse in Diffusion Models for Categorical Data

Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, Yuki Mitsufuji

Abstract: Latent diffusion models have enabled continuous-state diffusion models to handle a variety of datasets, including categorical data. However, most methods rely on fixed pretrained embeddings, limiting the benefits of joint training with the diffusion model. While jointly learning the embedding (via reconstruction loss) and the latent diffusion model (via score matching loss) could enhance performance, our analysis shows that end-to-end training risks embedding collapse, degrading generation quality. To address this issue, we introduce CATDM (also referred to as VQ-LCMD), a continuous diffusion framework within the embedding space that stabilizes training. We propose a novel objective combining the joint embedding-diffusion variational lower bound with a Consistency-Matching (CM) regularizer, alongside a shifted cosine noise schedule and random dropping strategy. The CM regularizer ensures the recovery of the true data distribution. Experiments on benchmarks show that CATDM mitigates embedding collapse, yielding superior results on FFHQ, LSUN Churches, and LSUN Bedrooms. In particular, CATDM achieves an FID of 6.81 on ImageNet $256\times256$ with 50 steps. It outperforms non-autoregressive models in machine translation and is on a par with previous methods in text generation.

Introduction

Generative modeling has seen a massive shift toward diffusion models, but handling discrete data—like the tokens produced by a Vector-Quantized Variational Autoencoder (VQ-VAE)—remains a complex challenge. While autoregressive models are common for discrete data, they are often slow and difficult to adapt for tasks like inpainting. Continuous-state diffusion models (CSDMs) offer a faster, more flexible alternative. However, most current latent diffusion methods rely on fixed, pretrained embeddings. Attempting to train the embeddings and the diffusion model simultaneously (joint training) often leads to embedding collapse, a failure mode where latent vectors converge to nearly identical points, destroying the model's representational capacity and resulting in poor generation quality.

The VQ-LCMD Framework

To address this, researchers from Sony AI and Stanford University introduced VQ-LCMD (Vector-Quantized Latent Consistency-Matching Diffusion). This framework enables the stable, joint training of discrete embeddings and continuous latent diffusion models by introducing several key architectural and algorithmic innovations.

The VQ-LCMD architecture showing the interaction between discrete and continuous spaces, the forward/reverse processes, and the integration of the Consistency-Matching loss.

Key Innovations

  1. Consistency-Matching (CM) Loss: This acts as a powerful regularizer. Based on the principles of Consistency Models, this loss ensures that the model’s predictions remain consistent across different points along the same Probability Flow ODE trajectory, effectively preventing the latent space from collapsing during training.
  2. Shifted Cosine Noise Schedule: Standard noise schedules can make denoising trivial at low noise levels if embedding norms are large. VQ-LCMD utilizes a shifted schedule to emphasize higher noise levels, forcing the model to learn global data structures rather than memorizing local shortcuts.
  3. Random Embedding Dropout: During training, the model randomly replaces tokens with a special [mask] token. This strategy prevents the model from relying on simple memorization and encourages it to capture deeper semantic relationships between different parts of the image.
  4. Simplified Final Loss: The framework moves away from complex Variational Lower Bound (VLB) weighting in favor of a simplified diffusion loss that distributes error more evenly across timesteps, improving training stability.

Experimental Evaluation

The researchers evaluated VQ-LCMD across several high-resolution image benchmarks, including FFHQ (faces), LSUN Churches, LSUN Bedrooms, and the ImageNet $256\times256$ dataset. The primary metrics used were Fréchet Inception Distance (FID), Precision, and Recall.

Quantitative Results

VQ-LCMD demonstrated significant improvements over both discrete-space and existing continuous-space diffusion baselines. On LSUN Bedrooms, it achieved an FID of 4.16, outperforming established models like VQ-Diffusion and MaskGIT. On ImageNet, the model reached an FID of 6.81 and an Inception Score (IS) of 225.31 using only 50 sampling steps.

Comparison of VQ-LCMD against discrete and continuous-space diffusion models across FFHQ and LSUN datasets.

Discussion and Insights

The most critical finding is the role of the Consistency-Matching loss in stabilizing joint training. Ablation studies revealed that without this loss, the model's FID on FFHQ plummeted from 7.25 to a disastrous 186.95, clearly illustrating the onset of embedding collapse. Furthermore, the results prove that trainable embeddings significantly outperform fixed ones; by allowing the latent space to evolve alongside the diffusion model, the system finds a more efficient representation tailored to the specific data distribution.

While VQ-LCMD represents a major step forward, future work may explore different architectural backbones beyond Transformers and extend the framework to other modalities such as text, music, and graphs. Additionally, further distillation could potentially reduce the required sampling steps even further, moving toward real-time generation.

Conclusion

VQ-LCMD successfully bridges the gap between discrete representations and continuous diffusion models. By mitigating embedding collapse through consistency matching and optimized noise scheduling, it enables high-quality, efficient image generation that sets new benchmarks for latent diffusion models.

Summary updated

Abstract

Latent diffusion models have enabled continuous-state diffusion models to handle a variety of datasets, including categorical data. However, most methods rely on fixed pretrained embeddings, limiting the benefits of joint training with the diffusion model. While jointly learning the embedding (via reconstruction loss) and the latent diffusion model (via score matching loss) could enhance performance, our analysis shows that end-to-end training risks embedding collapse, degrading generation quality. To address this issue, we introduce CATDM, a continuous diffusion framework within the embedding space that stabilizes training. We propose a novel objective combining the joint embedding-diffusion variational lower bound with a Consistency-Matching (CM) regularizer, alongside a shifted cosine noise schedule and random dropping strategy. The CM regularizer ensures the recovery of the true data distribution. Experiments on benchmarks show that CATDM mitigates embedding collapse, yielding superior results on FFHQ, LSUN Churches, and LSUN Bedrooms. In particular, CATDM achieves an FID of 6.81 on ImageNet $256\times256$ with 50 steps. It outperforms non-autoregressive models in machine translation and is on a par with previous methods in text generation.
BibTeX
@misc{nguyen2024mitigating,
  author = {Bac Nguyen and Chieh-Hsin Lai and Yuhta Takida and Naoki Murata and Toshimitsu Uesaka and Stefano Ermon and Yuki Mitsufuji},
  title = {{Mitigating Embedding Collapse in Diffusion Models for Categorical Data}},
  year = {2024},
  eprint = {2410.14758},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.14758},
}
Figure