SciRE-Solver: Accelerating Diffusion Models Sampling by Score-integrand Solver with Recursive Difference

Shigui Li, Wei Chen, Delu ZengSubmission to ICLR 2024 · Rejected

TL;DR

Diffusion models achieve state-of-the-art generative quality but suffer from slow sampling speeds due to the need for hundreds of neural network evaluations. SciRE-Solver accelerates continuous- and discrete-time diffusion ODE sampling down to 6–20 steps by integrating a concise score-integrand ODE formulation with a novel Recursive Difference (RD) method that estimates score function derivatives far more accurately than standard finite differences.

What Problem Does This Paper Address?

Sampling from diffusion models (DMs) can be formulated as solving a reverse-time ordinary differential equation (diffusion ODE). Standard numerical ODE solvers (e.g., Runge-Kutta, linear multi-step methods) typically require calculating derivatives of the drift term. In diffusion models, this drift term depends on a massive neural network (the score or noise prediction network $\epsilon_\theta(x, t)$), making exact Jacobian and higher-order derivative calculations computationally prohibitive.

Previous fast solvers (like DPM-Solver or DEIS) rely on finite difference (FD) approximations or exponential integrators. However, standard FD introduces noticeable truncation errors by abruptly dropping higher-order derivatives, leading to instability or over-rendering in few-step sampling regimes (e.g., $\le 15$ function evaluations). The authors tackle how to accurately and stably approximate score derivatives without additional model training or expensive computations.

Methods

The paper introduces two core mathematical concepts:

  1. Score-Integrand Formulation of Diffusion ODEs:
    Using the variation-of-constants formula, the authors transform the diffusion ODE into a simplified representation parameterized by the Noise-to-Signal Ratio ($\text{NSR}(t) = \sigma_t / \alpha_t$): $$\frac{x_t}{\alpha_t} - \frac{x_s}{\alpha_s} = \int_{\text{NSR}(s)}^{\text{NSR}(t)} \epsilon_\theta\big(x_{r_{\text{NSR}}(\tau)}, r_{\text{NSR}}(\tau)\big) \, d\tau$$ Because the integral isolates the score network evaluations without exponential-product terms, it proves empirically more stable under small step counts ($\le 10$). An equivalent formulation is derived for data-prediction models ($x_\theta$).

  2. Recursive Difference (RD) Method:
    Instead of truncating higher-order terms in the Taylor series expansion of $\epsilon_\theta$, the RD method recursively expresses higher-order derivatives in terms of lower-order differences across evaluation points. When simplified for practical algorithms, RD introduces a corrective scaling factor: $$\Gamma^{(1)}(\tau) \approx \frac{1}{\phi_1(m)} \frac{\Gamma(\tau_{t_{i-1}}) - \Gamma(\tau_{t_i})}{h_{t_i}}, \quad \text{where } \phi_1(m) = \sum_{k=1}^m \frac{(-1)^{k-1}}{k!}$$ For $m=3$, $\phi_1(3) = 2/3$, and as $m \to \infty$, $\phi_1(m) \to \frac{e-1}{e}$. This effectively amplifies and counterbalances the derivative estimate with residual higher-order information.

  3. SciRE-Solver Family:

  4. SciRE-V1 & SciRE-V2: 2nd- and 3rd-order single-step solvers with proven convergence order guarantees of at least $k-1$ (and empirically $k$) for noise- and data-prediction models, respectively.
  5. Multi-step Variants (SciRE-V1-2m, SciRE-V2-2m): Backward-difference versions that reuse previous network evaluations (caching) to reduce NFEs.
  6. SciRE-V1-agile: A dynamic schedule pairing 3rd-order, 2nd-order, and 1st-order (DDIM) steps to fully exploit arbitrary evaluation budgets.
  7. Custom Time Trajectories: Parametrized non-uniform schedules (NSR-type and Sigmoid-type) that allocate denser sampling points near the start and end of the diffusion trajectories.

Datasets

The authors evaluated SciRE-Solver using standard pre-trained generative benchmarks:

  • CIFAR-10 ($32 \times 32$): Discrete-time DDPM, continuous-time VP-deep, and EDM checkpoints.
  • CelebA ($64 \times 64$): Discrete-time model.
  • ImageNet ($64 \times 64$, $128 \times 128$, $256 \times 256$, and $512 \times 512$): Guided and unguided diffusion models.
  • LSUN Bedroom ($256 \times 256$).
  • Stable Diffusion: Qualitative text-to-image synthesis evaluated under challenging artistic prompts.

No new datasets were introduced.

Key Insights and Ideas

  • Why Finite Difference Falls Short: Standard FD sets truncation error to $O(h)$, discarding the curvature of the score function. Exponential integrators (like DPM-Solver) substitute $(e^h - 1)/h$, which acts as an amplification factor; while this improves performance in some regimes, it tends to over-render and distort images when step sizes $h$ are large.
  • RD as a Principled Compensator: RD mathematically derives the amplification factor $\frac{1}{\phi_1(m)}$ directly from the recursive structure of Taylor series remainders, allowing the solver to account for higher-order derivatives without explicitly computing them.
  • Score-Integrand Space vs. Exponential Integrators: Directly integrating over NSR space reduces numerical oscillations at small NFEs ($\le 10$), mitigating the color saturation artifacts often observed in other fast solvers.

Key Results

  • Continuous-time CIFAR-10 (VP-deep):
  • 3.48 FID at 12 NFE and 2.42 FID at 20 NFE (SciRE-V1-agile).
  • Achieves 2.40 FID at 100 NFE, surpassing the original 1000-step SDE baseline (2.41 FID).
  • Pretrained EDM on CIFAR-10:
  • 2.29 FID at 12 NFE and 1.76 FID at 100 NFE.
  • Discrete-time CIFAR-10 & CelebA 64:
  • On CIFAR-10, achieves 3.15 FID in 84 NFE, outperforming 1000-step DDPM (3.16 FID).
  • On CelebA $64 \times 64$, achieves 2.17 FID at 18 NFE and 2.02 FID at 50 NFE.
  • High-Resolution & Text-to-Image:
  • On ImageNet $128 \times 128$, SciRE-V1-2 reaches 3.67 FID at 20 NFE, outperforming DPM-Solver-2 at 50 NFE (4.17 FID).
  • Produces detailed, artifact-free Stable Diffusion generations with as few as 5 to 6 NFEs.

Limitations and Future Work

  • Limited Solver Order ($k \le 3$): Deriving and implementing RD-based solvers beyond order 3 involves combinatorial complexity and requires evaluating derivatives across multiple intermediate points, which was left for future work.
  • Simplified Truncation: The deployed algorithm uses a simplified RD approximation that drops higher-order derivative terms at point $\tau_t$; a generalized multi-point RD variant has not yet been developed.
  • Hyperparameter Sensitivity for Schedules: Optimal performance requires selecting between Sigmoid-type trajectories (best for $\le 15$ NFEs) and NSR-type trajectories ($> 15$ NFEs) and tuning schedule parameters ($k \in [2, 7]$). Automating trajectory selection remains an open problem.

Summary updated

Abstract

One downside of Diffusion models (DMs) is their slow iterative process. Recent algorithms for fast sampling are designed from the differential equations. However, in the fast algorithms, estimating the derivative of the score function evaluations becomes intractable due to the complexity of large-scale, well-trained neural networks. In this work, we introduce the recursive difference method to calculate the derivative of the score function networks. Building upon, we propose \emph{SciRE-Solver} with the convergence order guarantee for accelerating DMs sampling. Our proposed sampling algorithms attain SOTA FIDs in comparison to existing training-free sampling algorithms, under various number of score function evaluations (NFE). Such as, we achieve $3.48$ FID with $12$ NFE, and $2.42$ FID with $20$ NFE for continuous-time model on CIFAR-10; $1.79$ FID with $20$ NFE and $1.76$ FID with $100$ NFE for the pretrained model of EDM. Experiments demonstrate also that demonstrate that SciRE-Solver with multi-step methods can achieve high-quality samples on popular text-to-image generation tasks with only 6$\sim$20 NFEs.
BibTeX
@misc{li2023scire,
  author = {Shigui Li and Wei Chen and Delu Zeng},
  title = {{SciRE-Solver: Accelerating Diffusion Models Sampling by Score-integrand Solver with Recursive Difference}},
  year = {2023},
  eprint = {2308.07896},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.07896},
}

Loading related content...

Figure