Learning Diffusion Bridges on Constrained Domains

Xingchao Liu, Lemeng Wu, Mao Ye, qiang liuICLR 2023 · OpenReview

Introduction

Standard diffusion models are primarily designed for unconstrained Euclidean spaces, making them awkward to apply to data with "hard" boundaries, such as discrete categories or bounded intervals. To address this, researchers introduce $\Omega$-Bridges, a unified framework that leverages Doob’s $h$-transform to mathematically guarantee that a diffusion process will land within a specific constrained domain. This approach allows for high-quality generation of mixed-type tabular data, grid-constrained point clouds, and discrete images without relying on ad-hoc workarounds.

The Problem with Continuous Diffusion

Diffusion models have revolutionized generative AI, but they suffer from a "continuity" problem. Most are built on the assumption that data lives in an infinite, continuous space ($\mathbb{R}^d$). However, real-world data is often messy and constrained:

  • Discrete: Pixel values (0-255) or categorical labels (e.g., dog vs. cat).
  • Bounded: Human age (0-120) or normalized coordinates.
  • Mixed: Tabular data containing both continuous variables (salaries) and discrete attributes (occupations).

Previously, researchers handled these constraints using case-by-case hacks, such as dequantization (adding noise to discrete data to make it continuous) or specialized multinomial diffusions. These methods often lack a unified mathematical foundation and can produce sub-optimal results. The authors seek a single, elegant framework that works for any definable domain.

Methodology: The $\Omega$-Bridge Framework

The authors propose the $\Omega$-Bridge, a diffusion process designed to terminate exactly within a desired domain $\Omega$.

Illustration of the diffusion process constrained to land within specific target domains at time T.

The key innovation is the decomposition of the "drift"—the force that moves particles during diffusion—into two distinct parts:

  1. The Singular Force ($\eta^\Omega$): Derived using Doob’s $h$-transform from stochastic calculus. This is a fixed, non-learned force that acts like a magnetic pull toward the valid domain $\Omega$. As the process nears the end time $T$, this force becomes infinitely strong, ensuring the final sample is valid (e.g., an integer or a specific category).
  2. The Neural Force ($f^\theta$): A learnable neural network. While the singular force ensures the sample is valid, the neural force ensures the sample is realistic by matching the statistical distribution of the training data.

To train this model, the authors use a simplified version of Expectation-Maximization. Instead of complex posterior inference, they fit the model to "x-bridges"—trajectories that are conditioned to end at specific real data points $x$ from the training set.

Key Insights and Applications

The authors evaluated their framework across four diverse tasks: Adult Income (mixed tabular data), ShapeNet (3D point clouds), CityScapes (semantic segmentation), and CIFAR10 (discrete images).

  • Constraints as Conditioning: The paper reframes "constraints" as a conditioning problem. By treating the domain $\Omega$ as a target to be reached at time $T$, they can leverage the rigorous math of bridge processes.
  • Universal Approximation Power: A major insight is that if the neural network is powerful enough, the model doesn't need a complex "E-step" during training. It can simply learn to approximate the score of the data distribution relative to the singular force.
  • Geometric Uniformity: In 3D point clouds, the "Grid-Bridge" produced much more uniform point distributions than standard continuous models, leading to significantly better 3D mesh reconstructions.
Comparison of generated point clouds and reconstructed meshes, highlighting the superior geometric uniformity of the Grid-Bridge.

Key Results

  • Tabular Data: The "Mixed-Bridge" outperformed state-of-the-art GAN and VAE baselines (like CTGAN and TVAE) in classification accuracy when using synthetic data to train classifiers.
  • Semantic Segmentation: On CityScapes, the model achieved better Evidence Lower Bound (ELBO) scores than specialized categorical models like Argmax Flow and Multinomial Diffusion.
  • Discrete Images: For CIFAR10, the Integer-Bridge achieved a lower Frechet Inception Distance (FID) of 6.77, beating the specialized D3PM model (FID 7.34), proving that treating images as discrete integers is highly effective.

Limitations and Future Work

While highly effective, the current method works best when the domain $\Omega$ can be broken down into independent dimensions (product spaces). Applying this to more complex, non-factorizable manifolds (like the surface of a sphere or a specific 3D shape) remains a challenge. Additionally, the choice of the "baseline" process and the specific noise schedule has a large impact on performance and currently requires manual tuning. Finally, calculating the singular force for very high-dimensional discrete spaces can still be computationally intensive, though the authors provide efficient approximations for common cases.

Summary updated

Abstract

Diffusion models have achieved promising results on generative learning recently. However, because diffusion processes are most naturally applied on the unconstrained Euclidean space $\mathrm{R}^d$, key challenges arise for developing diffusion based models for learning data on constrained and structured domains. We present a simple and unified framework to achieve this that can be easily adopted to various types of domains, including product spaces of any type (be it bounded/unbounded, continuous/discrete, categorical/ordinal, or their mix). In our model, the diffusion process is driven by a drift force that is a sum of two terms: one singular force designed by $Doob's~ h$-$transform$ that ensures all outcomes of the process to belong to the desirable domain, and one non-singular neural force field that is trained to make sure the outcome follows the data distribution statistically. Experiments show that our methods perform superbly on generating tabular data, images, semantic segments and 3D point clouds.
BibTeX
@inproceedings{liu2023learning,
  author = {Xingchao Liu and Lemeng Wu and Mao Ye and qiang liu},
  title = {{Learning Diffusion Bridges on Constrained Domains}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=WH1yCa0TbB},
}
Figure