Joint Diffusion for Mixed Continuous and Discrete Data

This direction is split into other directions: joint discrete-continuous diffusion. Its last version stays readable and citable here.

Version 1 of 144 papers50 conceptsfrozen 2026-10-04

Most real-world data is messy and heterogeneous. A molecule is defined by both the continuous 3D coordinates of its atoms and the discrete categorical types of those atoms. A medical record contains continuous values like blood pressure alongside discrete diagnoses. While standard diffusion models have mastered continuous domains like images, they often struggle when forced to handle these mixed-type datasets.

The emerging research direction of Joint Diffusion addresses this by unifying disparate noise processes—typically Gaussian diffusion for continuous variables and multinomial or absorbing-state diffusion for discrete ones—into a single, synchronized generative framework.

The Heterogeneity Gap

In traditional generative modeling, researchers often handled mixed data by forcing everything into one format. One might "one-hot" encode categories and treat them as continuous vectors, or conversely, discretize continuous values into bins. Both approaches are flawed: continuous approximations of categorical data lead to "interpolation" between discrete identities (e.g., a model generating an atom that is 40% Carbon and 60% Nitrogen), while discretization destroys the precision of geometric or physical data.

Akim Kotelnikov and colleagues in TabDDPM demonstrated that treating these types natively is superior. By applying Gaussian noise to numerical features and multinomial noise (where a category "flips" to a random alternative) to categorical features, the model preserves the structural integrity of the data.

Synchronized Denoising via Shared Schedules

The core intuition behind joint diffusion is that the continuous and discrete components of an object are deeply interdependent. An object's identity (discrete) dictates its physical properties (continuous). For example, in 3D scene synthesis, the label "chair" implies a specific range of heights and orientations.

To capture these correlations, models like JointDiff, developed by Guillem Capellera and others, use a shared variance schedule. During the forward process, both the continuous trajectories and the discrete event states (like ball possession in sports) are corrupted simultaneously. During the reverse (denoising) process, the model predicts the clean version of both modalities at once. This ensures that as the model "decides" on a discrete state, it immediately refines the continuous coordinates to match that state, preventing the temporal or structural drift that occurs in sequential pipelines.

Co-evolution and Interacting Processes

A more sophisticated approach involves letting the two diffusion processes "talk" to each other. In the CoDi framework, Chaejeong Lee and Jayoung Kim propose co-evolving conditioning. Instead of a single joint model, they use two interacting processes where the denoising step for the continuous variables is conditioned on the current noisy state of the discrete variables, and vice versa.

This interaction is crucial for complex forecasting. Mai Zeng and Florence Regol introduced CDiff for event sequences, where the timing of an event (continuous) and its type (categorical) are modeled as coupled processes. This avoids the error accumulation typical of autoregressive models, as the entire sequence of times and types is refined together.

Applications Across Domains

The shift toward joint diffusion has unlocked state-of-the-art performance in several specialized fields:

  • Molecules: Emiel Hoogeboom and Victor Garcia Satorras pioneered the Equivariant Diffusion Model (EDM) (Equivariant Diffusion for Molecule Generation in 3D), which respects physical symmetries while jointly generating atom positions and types. Later, Xingang Peng and Jianzhu Ma extended this in MolDiff to include bond types, solving the "atom-bond inconsistency" problem.
  • Scientific Simulations: In particle physics and genomics, data is often "sparse"—containing many zeros that represent physical absence. Phil Ostheimer and Carl Herrmann developed Sparse Data Diffusion (SDD) (Sparse Data Diffusion for Scientific Simulations in Biology and Physics), which uses discrete "Sparsity Bits" diffused alongside continuous values to maintain sharp, physically accurate boundaries.
  • Medical Imaging: Yihao Liu and colleagues showed in MetaVoxel that a single diffusion process can model both high-dimensional 3D MRI scans (continuous) and clinical metadata like age and sex (discrete), allowing for zero-shot clinical prediction tasks.

Why It Works: The Power of Superposition

Perhaps the most intuitive explanation for the success of joint diffusion comes from SketchDNN by Sathvik Chereddy and John Femiani. They describe a Gaussian-Softmax diffusion where discrete labels exist in a state of "superposition" during the denoising process.

By allowing the model to maintain a probabilistic distribution over discrete categories while simultaneously adjusting continuous coordinates, the system can explore multiple structural possibilities at once. It only "collapses" into a specific category-coordinate pair as the noise level approaches zero, leading to much higher realism and diversity than models that fix categories early in the generation process.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_diffusion_for_mixed_continuous_and_b0054031,
  title = {Joint Diffusion for Mixed Continuous and Discrete Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-diffusion-for-mixed-continuous-and-discrete-data/rdr_b00540311b1e4791a52d6d57da04c13b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_diffusion_for_mixed_continuous_and_b0054031_v1,
  title = {Joint Diffusion for Mixed Continuous and Discrete Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-diffusion-for-mixed-continuous-and-discrete-data/rdr_b00540311b1e4791a52d6d57da04c13b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}