Continuous Diffusion Models for Text Generation

187 concepts124 papers

Traditional autoregressive language models generate text sequentially, predicting one token at a time from left to right. While highly effective, this incremental approach creates a computational bottleneck and limits the model's ability to perform global planning or revision.

Diffusion models offer an alternative paradigm: they generate an entire sequence in parallel by starting with a block of random noise and iteratively refining it. This non-autoregressive approach allows for global refinement and enables fine-grained, gradient-based control over complex linguistic structures, as demonstrated by Xiang Lisa Li et al. in Diffusion-LM Improves Controllable Text Generation. Because the generation process is iterative and global, researchers can inject complex semantic or lexical constraints directly into the generation trajectory using gradient guidance, a capability also explored by Sachin Kumar et al. in Gradient-based Constrained Sampling from Language Models and by Lei Sha in Gradient-guided Unsupervised Lexically Constrained Text Generation.

However, adapting diffusion models to language introduces a fundamental challenge: diffusion frameworks are natively designed for continuous data distributions (like images), whereas text is inherently discrete. Bridging this discrete-continuous modality gap is the central focus of this research direction.

The Discrete-Continuous Modality Gap

To apply continuous Gaussian diffusion to text, models typically map discrete vocabulary tokens into a continuous, high-dimensional embedding space. In the forward process, Gaussian noise is progressively added to these embedding vectors. In the reverse process, a neural network learns to denoise the corrupted vectors step-by-step, eventually mapping the refined continuous vectors back to discrete tokens via a rounding operation.

As surveyed by Kun Zhou et al. in Diffusion Models for Non-autoregressive Text Generation: A Survey, this setup creates a mathematical mismatch known as the rounding bottleneck. Because the diffusion model operates in a continuous space, its intermediate denoising trajectories often terminate in regions of the embedding space that do not correspond to any valid word in the vocabulary. When forced to map these intermediate vectors to the nearest valid token, the system suffers from severe rounding errors, frequently resulting in incoherent or nonsensical text.

Representation Collapse in High Dimensions

The rounding bottleneck is exacerbated by the geometry of raw word embedding spaces. Standard token embeddings are high-dimensional and highly sparse. When Gaussian noise is added to a word vector, it quickly drifts into vast, empty regions of the space where semantic relationships dissolve. Robin Strudel et al. note in Self-conditioned Embedding Diffusion for Text Generation that standard diffusion models struggle with this sparsity, making it difficult to achieve the same level of prompt alignment and coherence seen in image synthesis models.

Furthermore, raw embeddings lack contextual information. As discussed by Alexander Shabalin et al. in TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings, forcing a diffusion model to resolve complex linguistic ambiguities from uncontextualized embeddings places an overwhelming burden on the generator. Without structural modifications, diffusing directly on high-dimensional word embeddings often triggers representation collapse, as highlighted by Justin Lovelace et al. in Latent Diffusion for Language Generation.

Mechanisms for Bridging the Gap

To overcome these representational barriers, recent architectures employ three primary strategies to smooth the continuous-discrete interface:

1. Compressed Latent Spaces

Instead of diffusing raw token embeddings, models can compress variable-length text into a highly compact, continuous latent space derived from pre-trained language models. Viacheslav Meshchaninov et al. demonstrate in Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling that shrinking text representations while maintaining semantic richness allows standard continuous diffusion architectures to operate efficiently. This compression prevents the representation collapse typically caused by the high dimensionality of raw token spaces.

2. Pre-trained Decoders as Rounding Operators

Rather than relying on a naive nearest-neighbor lookup to project continuous vectors back into text, models can integrate pre-trained sequence-to-sequence architectures. Yimin Ou et al. show in Effective Integration of Text Diffusion and Pre-Trained Language Models with Linguistic Easy-First Schedule that replacing traditional rounding steps with a pre-trained decoder (such as BART) successfully bridges the continuous diffusion space and discrete text generation. The decoder leverages its inherent understanding of clean text spaces to map noisy continuous trajectories back into fluent language.

3. Simplex Diffusion

An alternative approach bypasses embedding spaces entirely by performing diffusion directly on the vocabulary logit simplex. Rabeeh Karimi mahabadi et al. introduce this concept in TESS: Text-to-Text Self-Conditioned Simplex Diffusion. By constraining the continuous trajectories within a probability simplex, the model avoids the need for external decoders or clamping methods, eliminating the primary source of rounding errors.

Discrete Diffusion and Hybrid Pathways

While continuous embedding diffusion adapts Gaussian noise to text, a parallel paradigm utilizes purely discrete diffusion. Instead of adding continuous noise, discrete diffusion models corrupt text sequences by replacing tokens with masks or random words, training the model to reconstruct the original tokens directly within the vocabulary space, as outlined by Kun Zhou et al.

Hybrid models attempt to combine the benefits of both worlds. For example, Shansan Gong et al. develop DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models, which incorporates a "soft absorbing state" into a continuous diffusion framework. This mechanism mimics discrete masking while remaining within a continuous embedding framework, allowing the model to retain gradient-flow advantages while significantly accelerating training and inference convergence.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_continuous_modality_gap_9a530a25,
  title = {discrete-continuous modality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-continuous-modality-gap/rdr_9a530a251a6d4dcbb2ed7db50d8ea98e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discrete_continuous_modality_gap_9a530a25_v1,
  title = {discrete-continuous modality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-continuous-modality-gap/rdr_9a530a251a6d4dcbb2ed7db50d8ea98e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake