Generative Fractional Diffusion Models

Gabriel Nobis, Maximilian Springenberg, Marco Aversa, Michael Detzel, Rembert Daems, Roderick Murray-Smith, Shinichi Nakajima, Sebastian Lapuschkin, Stefano Ermon, Tolga Birdal, Manfred Opper, Christoph Knochenhauer, Luis Oala, Wojciech Samek

TL;DR

Generative Fractional Diffusion Models (GFDM) introduce a novel continuous-time framework that replaces standard Brownian motion with fractional Brownian motion (fBM). By leveraging a Markovian approximation of fBM, the authors provide a way to control the "roughness" of diffusion paths, leading to significantly better image quality and diversity with minimal extra computational cost.

What Problem Does This Paper Address?

Standard score-based diffusion models rely on Brownian motion (BM) to transform data into noise. While successful, BM has two major theoretical limitations: it has independent increments (no memory) and is "light-tailed." In practice, this can lead to slow convergence, mode collapse on imbalanced datasets, and a lack of sample diversity.

The authors argue that the purely Markovian nature of BM might be too simplistic to capture the rich, correlated structures found in real-world data. They propose using fractional Brownian motion (fBM), a generalization of BM that features correlated increments and a tunable "Hurst index" ($H$). However, fBM is notoriously difficult to work with because it is non-Markovian and not a "semimartingale," meaning standard stochastic calculus (and the usual reverse-time diffusion math) doesn't apply directly.

Methods

The core innovation of GFDM is making fractional diffusion mathematically tractable for generative modeling.

  1. Markov Approximation (MA-fBM): Instead of using "true" fBM, the authors approximate it as a linear combination of $K$ Ornstein-Uhlenbeck (OU) processes. This "Markovianization" allows them to use standard tools while still capturing the long-range dependencies of fBM.
  2. Augmented Forward Process: The system is modeled by an augmented state vector $Z_t$ that includes both the data $X_t$ and the $K$ hidden OU processes.
  3. Augmented Score Matching: Training a model on the full augmented state would be computationally expensive. The authors derive a clever "augmented score matching" loss. This allows them to train a neural network that only predicts the score for the original data dimensions, while the scores for the augmenting processes are handled in closed form.
  4. Hurst Index Control: The parameter $H \in (0, 1)$ acts as a dial for path behavior:
  5. $H = 0.5$: Recovers standard Brownian motion.
  6. $H < 0.5$: Produces "rough" paths (sub-diffusion).
  7. $H > 0.5$: Produces "smooth" paths with long-term memory (super-diffusion).
Figure 1: Each data dimension transitions to a known prior distribution through a forward process that approximates a fractional diffusion process. The Hurst index H on the LHS interpolates between the roughness of a Brownian driven SDE and the underlying integration in PF ODEs. The driving noise process is a linear combination of the correlated processes on the RHS, all driven by the same Brownian motion. The score function of these augmenting processes is available in closed form and serves as guidance for the unknown score function.

Datasets

The authors evaluated GFDM on two standard image benchmarks:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of 60,000 32x32 color images in 10 classes.

Key Insights and Ideas

  • Super-diffusion is Superior for Images: The most significant finding is that the super-diffusive regime ($H > 0.5$) consistently outperforms standard diffusion. Specifically, $H=0.9$ yielded the best results, suggesting that the long-term memory and smoother paths help the model learn better data distributions.
  • Diversity through Correlation: By varying $H$ and the number of augmenting processes ($K$), researchers gain a new way to control sample diversity. Higher $K$ values were shown to increase pixel-wise diversity.
  • Efficiency: Despite the complex underlying math, GFDM requires the same number of score model evaluations as traditional models. The overhead for managing the augmenting processes is negligible (at most 11/1000ths of a second per training step).

Key Results

GFDM achieves state-of-the-art results compared to purely Brownian-driven baselines:

  • MNIST: Improved FID from 1.44 (standard) to 0.72 (GFDM).
  • CIFAR-10: Improved FID from 4.85 (standard) to 3.77 (GFDM).
  • Faster Convergence: As shown in Figure 2, GFDM reaches lower FID scores with fewer sampling steps (NFEs) than standard models.
Figure 2: Comparison of the super-diffusive regime and purely Brownian dynamics in terms of average FID over three rounds of sampling plotted across different NFEs.

The quantitative benefits are summarized in the following table, highlighting that the best scores (boxed) are consistently found in the super-diffusive $H=0.9$ regime.

Table 2: Quantitative results on MNIST and CIFAR-10 comparing BM-driven vs MA-fBM-driven models.

Limitations and Future Work

  • Dataset Scope: The current evaluation is limited to relatively low-resolution image datasets (MNIST, CIFAR-10). Testing on high-resolution images or different modalities like audio or 3D structures remains for future work.
  • Optimal Hurst Index: The paper finds $H=0.9$ is best for images, but the optimal $H$ likely depends on the specific data modality.
  • Theoretical Limits: An open question remains regarding the behavior of the model as the number of augmenting processes $K$ approaches infinity, which would converge to "true" non-Markovian fractional Brownian motion.
  • Rough Time Series: The authors suggest that the "rough" regime ($H < 0.5$) might be particularly useful for generating financial or scientific time-series data, which often exhibits such characteristics.

Summary updated

Abstract

We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in generative fractional diffusion models (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose augmented score matching to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of rough paths whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models
BibTeX
@inproceedings{nobis2024generative,
  author = {Gabriel Nobis and Maximilian Springenberg and Marco Aversa and Michael Detzel and Rembert Daems and Roderick Murray-Smith and Shinichi Nakajima and Sebastian Lapuschkin and Stefano Ermon and Tolga Birdal and Manfred Opper and Christoph Knochenhauer and Luis Oala and Wojciech Samek},
  title = {{Generative Fractional Diffusion Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=B9qg3wo75g},
}
Figure