Abstract
Diffusion models have shown remarkable success in continuous domains like image and audio synthesis. However, applying them to discrete text generation via embedding-space diffusion faces significant challenges, namely embedding collapse and model degeneration. We present Difformer, a Transformer-based diffusion model that addresses these issues through two novel mechanisms: Anchor Loss and Noise Rescaling. Anchor Loss prevents embedding collapse by enforcing a robust, isotropic embedding space, while Noise Rescaling mitigates model degeneration by amplifying noise levels to ensure the model learns complex linguistic dependencies. Our experiments across multiple NLP benchmarks, including translation and summarization, demonstrate that Difformer achieves state-of-the-art performance, outperforming previous diffusion baselines and offering significant inference speedups over autoregressive models.
Introduction
The application of diffusion models to text generation typically involves mapping discrete tokens into a continuous embedding space. While this allows for the use of standard diffusion frameworks, it introduces two primary optimization hurdles. First, embedding collapse occurs when the model learns to map different tokens to nearly identical vectors to simplify the denoising task, leading to training instability. Second, model degeneration arises in high-dimensional spaces where standard noise schedules fail to provide sufficient variance. In such cases, the denoising task becomes trivial—the model simply identifies the nearest neighbor without considering context—resulting in "lazy" models that ignore input sequences.
Methods
Difformer addresses these challenges with a Transformer-based architecture incorporating two key innovations designed to stabilize training and improve sequence modeling.
Anchor Loss
To combat embedding collapse, Difformer introduces Anchor Loss. Unlike traditional rounding losses that merely encourage embeddings to stay near their discrete counterparts, Anchor Loss applies a penalty based on the model's prediction of the clean data ($\hat{z}_0$). By forcing the model to map even highly inaccurate predictions back to correct "anchor" tokens, it acts as a "stress test" for the embedding space, ensuring it remains well-distributed (isotropic) and informative.
Noise Rescaling
To prevent model degeneration, the authors utilize the DeGeneration Score (DGS), which quantifies how easily a model can predict a token from a single noisy embedding. Difformer employs a rescaling factor ($F$) to amplify noise levels, ensuring the DGS remains low enough to force the model to learn global dependencies rather than local shortcuts. This "Noise Rescaling" framework ensures that the diffusion process remains challenging enough to drive meaningful learning.
Results
Difformer was evaluated on several benchmarks, including WMT14 En-De, Gigaword, and Quora Question Pairs. It consistently outperformed previous embedding diffusion baselines. For instance, on the IWSLT14 De-En translation task, Difformer achieved a BLEU score of 34.48, significantly higher than DiffuSeq's 28.78.
Furthermore, Difformer demonstrated competitive performance with standard autoregressive Transformers while providing up to a 6x speedup during inference by using parallel decoding with fewer steps (e.g., $K=20$). Qualitative results also indicate that Difformer produces more diverse and creative text compared to iterative non-autoregressive models like CMLM, which tend to be more repetitive.
Conclusion
Difformer successfully adapts diffusion models for text generation by addressing the fundamental flaws of embedding-space diffusion. While it requires longer training times compared to some non-autoregressive baselines and currently relies on an offline search for the rescaling factor, its performance and efficiency gains mark a significant step forward. Future work will explore dynamic rescaling factors and scaling these techniques to large language models.
Summary updated