A Score-Based Deterministic Diffusion Algorithm with Smooth Scores for General Distributions

Karthik Elamvazhuthi, Xuechen Zhang, Matthew Jacobs, Samet Oymak, Fabio Pasqualetti

Introduction

Standard diffusion models, such as DDPM and Stable Diffusion, have revolutionized generative modeling by learning to reverse a stochastic forward process that adds Gaussian noise to data. However, these models rely on the assumption that the data distribution has a Lipschitz logarithmic gradient—meaning the "score function" (the gradient of the log-density) must remain bounded.

In practice, real-world data often has compact support or resides on low-dimensional manifolds, where the probability density drops to zero outside specific boundaries. In these cases, the score function explodes as the density approaches zero, leading to numerical instability and requiring heuristic workarounds like early stopping. This paper introduces a deterministic diffusion framework designed to maintain smooth, bounded scores even for distributions with sharp boundaries.

Methodology: The Blob Method

The core innovation of this work is the replacement of the stochastic (SDE-based) forward process with a deterministic diffusion process based on particle repulsion, known as the "blob method."

  1. McKean-Vlasov Process: Instead of adding random noise, particles move according to a mutually repelling force. This movement is governed by a kernel $K$ (typically a Gaussian kernel).
  2. Smooth Vector Fields: By defining particle movement through a convolution with a smooth kernel ($ abla K * \rho$), the resulting vector field is guaranteed to be Lipschitz-continuous. This ensures that the score never explodes, regardless of the underlying data distribution's geometry.

  3. Porous Media Equation (PME): The authors establish a link between this process and the Porous Media Equation from physics. As the kernel width $\epsilon$ approaches zero, the forward process converges to a simple noise distribution, such as a uniform distribution on a bounded set.

  4. Deterministic Reverse Flow: Because the forward trajectories are deterministic and differentiable, the reverse sampling process is more regular and easier for neural networks to learn than the jagged paths associated with Brownian motion.

Experimental Setup

The researchers evaluated the proposed deterministic diffusion algorithm against standard stochastic baselines across several challenging datasets:

  • Synthetic Data: 1D intervals and 2D circles, which serve as perfect test cases for compact support where density is strictly zero outside the shape.
  • Binary MNIST: Hand-written digits where the background pixels have zero density.
  • Padded CIFAR-10: $32 \times 32$ images padded to $40 \times 40$ with black pixels to test how models handle the transition from data-rich regions to empty space.
Generative performance on 1D interval data: (left) Target, (center) Stochastic, (right) Deterministic.

Key Results and Insights

The deterministic approach demonstrates significant improvements in both stability and generative fidelity. Theoretical analysis shows that this method has a linear dependence on score-matching error, making it more robust to imperfectly trained neural networks than stochastic models, which typically exhibit a square-root dependence.

Quantitative Performance

  • 2D Circle Fidelity: On the 2D circle dataset, the deterministic model achieved a Wasserstein distance of 2.057, vastly outperforming the stochastic model's 4.628.
  • CIFAR-10 Quality: On the padded CIFAR-10 dataset, the deterministic approach reached an FID of 27.93, compared to 42.05 for the stochastic baseline.
Comparison of generative performance on a 2D circle distribution: (a) Target, (b) Stochastic, (c) Deterministic.

Qualitative Observations

Visual results on MNIST and CIFAR-10 highlight the model's ability to produce cleaner backgrounds and sharper edges. While stochastic models often leave noisy artifacts in zero-density regions due to exploding scores, the deterministic model remains stable, resulting in much higher visual quality in bounded domains.

Conclusion and Future Work

By leveraging particle repulsion and smooth vector fields, this deterministic diffusion algorithm effectively addresses the instability issues inherent in standard score-based models for distributions with compact support. Future research will focus on optimizing kernel selection for specific data types, characterizing long-term convergence for fixed kernel widths, and scaling these particle-based methods for high-resolution image generation.

Summary updated

Abstract

Score matching based diffusion has shown to achieve the state of art results in generation modeling. In the original score matching based diffusion algorithm, the forward equation is a differential equation for which the probability density equation evolves according to a linear partial differential equation, the Fokker-Planck equation. A drawback of this approach is that one needs the data distribution to have a Lipschitz logarithmic gradient. This excludes a large class of data distributions that have a compact support. We present a deterministic diffusion process for which the vector fields are always Lipschitz and hence the score does not explode for probability measures with compact support. This deterministic diffusion process can be seen as a regularization of the porous media equation equation, which enables one to guarantee long term convergence of the forward process to the noise distribution. Though the porous media equation is itself not always guaranteed to have a Lipschitz vector field, it can be used to understand the closeness of the output of the algorithm to the data distribution as a function of the the time horizon and score matching error. This analysis enables us to show that the algorithm has better dependence on the score matching error than approaches based on stochastic diffusions. Using numerical experiments we verify our theoretical results on example one and two dimensional data distributions which are compactly supported. Additionally, we validate the approach on a modified MNIST data set for which the distribution is concentrated on a compact set. In each of the experiments, the approach using deterministic diffusion performs better that the diffusion algorithm with stochastic forward process, when considering the FID scores of the generated samples.
BibTeX
@inproceedings{elamvazhuthi2024score,
  author = {Karthik Elamvazhuthi and Xuechen Zhang and Matthew Jacobs and Samet Oymak and Fabio Pasqualetti},
  title = {{A Score-Based Deterministic Diffusion Algorithm with Smooth Scores for General Distributions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29072},
}

Loading related content...

Figure