Joint Discrete and Continuous Diffusion

27 concepts23 papers1 proposal

Standard denoising diffusion models excel when operating on a single, homogeneous data type. In continuous spaces, they generate photorealistic images and high-fidelity audio by gradually shaving off Gaussian noise. In discrete spaces, they generate coherent text or code by iteratively unmasking tokens.

However, the real world is rarely homogeneous. Consider a molecule: it consists of continuous 3D spatial coordinates paired with discrete atom types and bond categories. Consider a graphic layout: it contains continuous bounding box coordinates alongside discrete element labels (e.g., "header", "image"). Or consider tabular data, which routinely mixes continuous financial figures with categorical identifiers.

To generate such heterogeneous data, we cannot simply treat the modalities in isolation. If we generate the structure first and the labels second, the labels will not fit the geometry. If we map discrete categories into a continuous space, the model wastes capacity learning to avoid the meaningless voids between categories.

Joint discrete-continuous diffusion solves this by running distinct, specialized noise processes in parallel, while binding them together through a single, shared denoising network.


The Core Mechanism: Parallel Corruption, Unified Denoising

To understand why joint diffusion works, we must look at how it structures the forward (corruption) and reverse (denoising) pathways. Instead of forcing heterogeneous data into a single mathematical representation, the framework applies a tailored corruption process to each data type.

Forward Process (Adding Noise):
Continuous Features (e.g., Coordinates)  ---> Add Gaussian Noise --------\
                                                                           ---> Combined Noisy State
Discrete Features   (e.g., Categories)   ---> Flip/Mask (Multinomial) ----/

Reverse Process (Denoising):
Combined Noisy State ---> [ Shared Neural Network ] ---> Predict Clean Features Simultaneously

1. The Forward Process: Tailored Destruction

For a mixed dataset containing continuous features $x_c$ and discrete features $x_d$, the forward process injects noise into both components independently at each timestep $t$:

  • Continuous components undergo standard Gaussian diffusion, where values are incrementally nudged by small random vectors.
  • Discrete components undergo categorical corruption. This is typically achieved via multinomial diffusion—where a category has a small probability of flipping to a random alternative state, as utilized by Kotelnikov et al. in TabDDPM—or via absorbing-state diffusion, where tokens are progressively replaced by a special [MASK] token, a strategy favored by Levi et al. in DLT.

2. The Reverse Process: Co-dependent Reconstruction

The magic of joint diffusion happens during the reverse process. Instead of using separate networks to denoise the continuous and discrete parts, a single neural network—typically a Transformer or a Graph Neural Network (GNN)—receives the noisy versions of both modalities simultaneously.

At timestep $t$, the network looks at the blurry continuous coordinates and the corrupted discrete labels. Because the network must predict the clean versions of both modalities at once, it is forced to learn their cross-modal dependencies.


Why It Works: The Intuition of Mutual Constraint

Why does this parallel approach outperform multi-stage generation or continuous embeddings? The answer lies in how information flows during the reverse sampling trajectory.

During the early stages of the reverse process (high $t$), the data is mostly noise. The model cannot yet determine exact spatial coordinates or precise categorical identities. Instead, it establishes global, coarse-grained relationships. For instance, in layout generation, the model might decide that a large block exists in the center of the page before it settles on whether that block is an "image" or a "paragraph".

As the noise level decreases, the continuous and discrete features act as mutual constraints:

  • Geometry guides semantics: If the continuous coordinates begin to resolve into a long, thin horizontal strip, the shared network leverages this geometric cue to increase the probability that the corresponding discrete label is a "headline".
  • Semantics guide geometry: Conversely, if the discrete diffusion process resolves an atom type to "Oxygen", the network immediately constrains the continuous spatial denoising step to respect the physical bond lengths and angles typical of oxygen atoms.

By allowing both modalities to co-evolve across the diffusion timeline, the model avoids the compounding errors common in two-stage pipelines (where a mistake in the first stage dooms the second).


Diverse Architectures for Diverse Domains

The power of joint diffusion has led to its rapid adoption across several distinct domains, each adapting the core principle to its unique structural constraints.

1. 3D Molecules and Graphs

In molecular generation, physical validity requires that spatial positions align perfectly with chemical identities. Hoogeboom et al. demonstrated this in Equivariant Diffusion for Molecule Generation in 3D (EDM) by jointly diffusing 3D coordinates and atom types while maintaining physical equivariance.

Building on this, Vignac et al. introduced MiDi, which incorporates discrete bond types (the molecular graph skeleton) alongside continuous 3D coordinates into a single end-to-end differentiable process. To prevent the model from interpolating between invalid intermediate chemical identities, Le et al. leveraged hybrid continuous-categorical state spaces in their EQGAT-diff framework, significantly boosting structural validity.

2. Tabular Data Synthesis

Tabular data lacks spatial structure but possesses intense feature correlations. Kotelnikov et al. (TabDDPM) and Villaizán-Vallelado et al. (MTabGen) proved that combining Gaussian and multinomial diffusion allows models to capture complex correlations between numerical columns (like income) and categorical columns (like education level). To refine this further, Shi et al. developed TabDiff, which embeds these mixed operations within a unified continuous-time diffusion process using feature-wise learnable noise schedules.

3. Spatio-Temporal and Event Sequences

Joint diffusion is also highly effective for modeling sequences where actions and timing are intertwined. In multi-agent trajectory generation, Capellera et al. introduced JointDiff, which couples continuous player coordinates with discrete event states (such as ball possession) during the denoising phase to ensure physical coherence.

Similarly, for Temporal Point Processes (TPPs)—where events occur at irregular intervals—models must capture both when something happens (continuous time) and what happens (discrete event type). Zeng (CDiff) and Mukherjee (ADiff4TPP) demonstrate that treating time and event categories as interacting diffusion processes avoids the error accumulation that typically plagues autoregressive forecasting models.


Mathematical Unification: The Heat Equation Perspective

While running Gaussian and categorical diffusion in parallel works well in practice, researchers have also sought a unified mathematical foundation for this hybrid generation.

A compelling perspective comes from Schröder et al. in their work on Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces. They frame the diffusion of both continuous and discrete variables as solving a single heat equation across a structured, hybrid space. In this view, continuous variables diffuse along a smooth manifold, while discrete variables diffuse across the edges of a discrete graph. By unifying these processes under a single differential equation, the underlying model can learn a seamless joint distribution across highly heterogeneous feature spaces.

Key Foundational Work

DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformer (2023) — contributed 2 concept elements to this direction.
Equivariant Diffusion for Molecule Generation in 3D (2022) — contributed 2 concept elements to this direction.
TabDDPM: Modelling Tabular Data with Diffusion Models (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_discrete_continuous_diffusion_9f447b35,
  title = {joint discrete-continuous diffusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-discrete-continuous-diffusion/rdr_9f447b3500684dc6872918c67c281bcf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_discrete_continuous_diffusion_9f447b35_v1,
  title = {joint discrete-continuous diffusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-discrete-continuous-diffusion/rdr_9f447b3500684dc6872918c67c281bcf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake