Difformer: Empowering Diffusion Model on Embedding Space for Text Generation

Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, Linli Xu

Abstract

Diffusion models have shown remarkable success in continuous domains like image and audio synthesis. However, applying them to discrete text generation via embedding-space diffusion faces significant challenges, namely embedding collapse and model degeneration. We present Difformer, a Transformer-based diffusion model that addresses these issues through two novel mechanisms: Anchor Loss and Noise Rescaling. Anchor Loss prevents embedding collapse by enforcing a robust, isotropic embedding space, while Noise Rescaling mitigates model degeneration by amplifying noise levels to ensure the model learns complex linguistic dependencies. Our experiments across multiple NLP benchmarks, including translation and summarization, demonstrate that Difformer achieves state-of-the-art performance, outperforming previous diffusion baselines and offering significant inference speedups over autoregressive models.

Introduction

The application of diffusion models to text generation typically involves mapping discrete tokens into a continuous embedding space. While this allows for the use of standard diffusion frameworks, it introduces two primary optimization hurdles. First, embedding collapse occurs when the model learns to map different tokens to nearly identical vectors to simplify the denoising task, leading to training instability. Second, model degeneration arises in high-dimensional spaces where standard noise schedules fail to provide sufficient variance. In such cases, the denoising task becomes trivial—the model simply identifies the nearest neighbor without considering context—resulting in "lazy" models that ignore input sequences.

Methods

Difformer addresses these challenges with a Transformer-based architecture incorporating two key innovations designed to stabilize training and improve sequence modeling.

The Difformer framework illustrating the forward and reverse diffusion processes, including the Anchor Loss and Noise Rescaling components.

Anchor Loss

To combat embedding collapse, Difformer introduces Anchor Loss. Unlike traditional rounding losses that merely encourage embeddings to stay near their discrete counterparts, Anchor Loss applies a penalty based on the model's prediction of the clean data ($\hat{z}_0$). By forcing the model to map even highly inaccurate predictions back to correct "anchor" tokens, it acts as a "stress test" for the embedding space, ensuring it remains well-distributed (isotropic) and informative.

Noise Rescaling

To prevent model degeneration, the authors utilize the DeGeneration Score (DGS), which quantifies how easily a model can predict a token from a single noisy embedding. Difformer employs a rescaling factor ($F$) to amplify noise levels, ensuring the DGS remains low enough to force the model to learn global dependencies rather than local shortcuts. This "Noise Rescaling" framework ensures that the diffusion process remains challenging enough to drive meaningful learning.

Empirical analysis of noise rescaling: (a) BLEU scores over time steps, (b) DeGeneration Score (DGS) for different schedules, and (c) the effect of the rescaling factor F on the noise schedule.

Results

Difformer was evaluated on several benchmarks, including WMT14 En-De, Gigaword, and Quora Question Pairs. It consistently outperformed previous embedding diffusion baselines. For instance, on the IWSLT14 De-En translation task, Difformer achieved a BLEU score of 34.48, significantly higher than DiffuSeq's 28.78.

Furthermore, Difformer demonstrated competitive performance with standard autoregressive Transformers while providing up to a 6x speedup during inference by using parallel decoding with fewer steps (e.g., $K=20$). Qualitative results also indicate that Difformer produces more diverse and creative text compared to iterative non-autoregressive models like CMLM, which tend to be more repetitive.

Conclusion

Difformer successfully adapts diffusion models for text generation by addressing the fundamental flaws of embedding-space diffusion. While it requires longer training times compared to some non-autoregressive baselines and currently relies on an offline search for the rescaling factor, its performance and efficiency gains mark a significant step forward. Future work will explore dynamic rescaling factors and scaling these techniques to large language models.

Summary updated

BibTeX
@inproceedings{gao2024difformer,
  author = {Zhujin Gao and Junliang Guo and Xu Tan and Yongxin Zhu and Fang Zhang and Jiang Bian and Linli Xu},
  title = {{Difformer: Empowering Diffusion Model on Embedding Space for Text Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.261},
}
Figure