Introduction
Standard diffusion models, such as DDPM and Stable Diffusion, have revolutionized generative modeling by learning to reverse a stochastic forward process that adds Gaussian noise to data. However, these models rely on the assumption that the data distribution has a Lipschitz logarithmic gradient—meaning the "score function" (the gradient of the log-density) must remain bounded.
In practice, real-world data often has compact support or resides on low-dimensional manifolds, where the probability density drops to zero outside specific boundaries. In these cases, the score function explodes as the density approaches zero, leading to numerical instability and requiring heuristic workarounds like early stopping. This paper introduces a deterministic diffusion framework designed to maintain smooth, bounded scores even for distributions with sharp boundaries.
Methodology: The Blob Method
The core innovation of this work is the replacement of the stochastic (SDE-based) forward process with a deterministic diffusion process based on particle repulsion, known as the "blob method."
- McKean-Vlasov Process: Instead of adding random noise, particles move according to a mutually repelling force. This movement is governed by a kernel $K$ (typically a Gaussian kernel).
-
Smooth Vector Fields: By defining particle movement through a convolution with a smooth kernel ($ abla K * \rho$), the resulting vector field is guaranteed to be Lipschitz-continuous. This ensures that the score never explodes, regardless of the underlying data distribution's geometry.
-
Porous Media Equation (PME): The authors establish a link between this process and the Porous Media Equation from physics. As the kernel width $\epsilon$ approaches zero, the forward process converges to a simple noise distribution, such as a uniform distribution on a bounded set.
- Deterministic Reverse Flow: Because the forward trajectories are deterministic and differentiable, the reverse sampling process is more regular and easier for neural networks to learn than the jagged paths associated with Brownian motion.
Experimental Setup
The researchers evaluated the proposed deterministic diffusion algorithm against standard stochastic baselines across several challenging datasets:
- Synthetic Data: 1D intervals and 2D circles, which serve as perfect test cases for compact support where density is strictly zero outside the shape.
- Binary MNIST: Hand-written digits where the background pixels have zero density.
- Padded CIFAR-10: $32 \times 32$ images padded to $40 \times 40$ with black pixels to test how models handle the transition from data-rich regions to empty space.
Key Results and Insights
The deterministic approach demonstrates significant improvements in both stability and generative fidelity. Theoretical analysis shows that this method has a linear dependence on score-matching error, making it more robust to imperfectly trained neural networks than stochastic models, which typically exhibit a square-root dependence.
Quantitative Performance
- 2D Circle Fidelity: On the 2D circle dataset, the deterministic model achieved a Wasserstein distance of 2.057, vastly outperforming the stochastic model's 4.628.
- CIFAR-10 Quality: On the padded CIFAR-10 dataset, the deterministic approach reached an FID of 27.93, compared to 42.05 for the stochastic baseline.
Qualitative Observations
Visual results on MNIST and CIFAR-10 highlight the model's ability to produce cleaner backgrounds and sharper edges. While stochastic models often leave noisy artifacts in zero-density regions due to exploding scores, the deterministic model remains stable, resulting in much higher visual quality in bounded domains.
Conclusion and Future Work
By leveraging particle repulsion and smooth vector fields, this deterministic diffusion algorithm effectively addresses the instability issues inherent in standard score-based models for distributions with compact support. Future research will focus on optimizing kernel selection for specific data types, characterizing long-term convergence for fixed kernel widths, and scaling these particle-based methods for high-resolution image generation.
Summary updated