GENIE: Higher-Order Denoising Diffusion Solvers

Tim Dockhorn, Arash Vahdat, Karsten Kreis

TL;DR

GENIE (Higher-Order Denoising Diffusion Solvers) accelerates image synthesis in diffusion models by using a novel second-order ODE solver based on truncated Taylor methods. By distilling complex higher-order gradients into a tiny, efficient neural network "head," GENIE achieves state-of-the-art image quality with significantly fewer steps than traditional solvers while maintaining the ability to perform tasks like image encoding and guided sampling.

The "Speed Tax" of Diffusion Models

Denoising Diffusion Models (DDMs) have become the gold standard for generative AI, powering everything from Midjourney to Stable Diffusion. However, they come with a major "speed tax." To generate an image, these models must solve a complex differential equation (DE) by taking many tiny, iterative steps—often hundreds or thousands—to transform random noise into a coherent image.

Existing fast samplers like DDIM are essentially "first-order" solvers (akin to Euler's method). They assume the path from noise to data is a series of straight lines. Because the actual generative path is curved, these solvers require many steps to stay on track. While some researchers have tried higher-order solvers to capture this curvature, they usually rely on slow approximations or "black-box" distillation methods that break the underlying mathematical structure of the model, making it impossible to do things like "reverse" the process to encode an existing image.

Introducing GENIE

The authors introduce GENIE (Higher-Order Denoising Diffusion Solvers). The core technical contribution is a solver based on Truncated Taylor Methods (TTM). Specifically, they derive a second-order solver that uses the local curvature (the Jacobian) of the model's "score function" to take much larger, more accurate steps.

Linear vs. Quadratic Extrapolation

The "aha moment" is that the generative path of a diffusion model is highly curved, especially at the beginning of the process. By capturing this curvature via Taylor expansion (quadratic extrapolation), the solver can "see around the corner" and take much larger steps without drifting off the manifold, unlike standard linear extrapolation.

Overcoming the Computation Bottleneck

The main challenge is that computing second-order gradients is computationally expensive. To solve this, the authors use a clever two-step approach:

  1. Automatic Differentiation: They use auto-diff to extract Jacobian-vector products (JVPs) from the original model.
  2. Gradient Distillation: Instead of calculating these JVPs every time you want to generate an image, they train a small additional neural network head to predict these higher-order terms.
GENIE Architecture with Distillation Head

At inference time, you simply run the main model plus this tiny head (which adds less than 2% overhead for some models), giving you the accuracy of a second-order solver at nearly the speed of a first-order one. Unlike methods that distill the entire sampling process into a single-step GAN (which is hard to train and loses the ODE structure), GENIE distills only the higher-order terms. This keeps the "true" generative ODE intact.

Key Results

The researchers validated GENIE across several standard image generation benchmarks, including CIFAR-10, LSUN Bedrooms, ImageNet, and AFHQv2 (Cats). GENIE consistently outperformed popular solvers like DDIM and PNDM across all tested Number of Function Evaluations (NFEs).

  • Few-Step Dominance: At 10 steps on CIFAR-10, GENIE achieved an FID (Fréchet Inception Distance, where lower is better) of 5.28, compared to DDIM’s 11.2.
  • High-Resolution Upsampling: On the 512x512 "Cats" dataset, GENIE produced high-quality images in just 5 steps, outperforming all other fast samplers.
  • Preserved Functionality: Because GENIE solves the actual ODE, the authors demonstrated it could still perform image encoding/decoding (mapping an image back to noise and back again) and classifier-free guidance with much higher fidelity than DDIM at low step counts.

Limitations and Future Work

While the distillation "head" is small, it isn't entirely free. Depending on the architecture, it adds between 1.5% and 14.4% more computation per step. The authors accounted for this by comparing models based on total "compute budget" rather than just raw step counts. Additionally, unlike "plug-and-play" solvers, GENIE requires a short distillation training phase (around 50k iterations) for the additional head.

Looking ahead, this paper focused on second-order solvers. The authors suggest that exploring third-order or even higher-order TTMs could yield even more acceleration, though the distillation process might become more complex.

Summary updated

Abstract

Denoising diffusion models (DDMs) have emerged as a powerful class of generative models. A forward diffusion process slowly perturbs the data, while a deep model learns to gradually denoise. Synthesis amounts to solving a differential equation (DE) defined by the learnt model. Solving the DE requires slow iterative solvers for high-quality generation. In this work, we propose Higher-Order Denoising Diffusion Solvers (GENIE): Based on truncated Taylor methods, we derive a novel higher-order solver that significantly accelerates synthesis. Our solver relies on higher-order gradients of the perturbed data distribution, that is, higher-order score functions. In practice, only Jacobian-vector products (JVPs) are required and we propose to extract them from the first-order score network via automatic differentiation. We then distill the JVPs into a separate neural network that allows us to efficiently compute the necessary higher-order terms for our novel sampler during synthesis. We only need to train a small additional head on top of the first-order score network. We validate GENIE on multiple image generation benchmarks and demonstrate that GENIE outperforms all previous solvers. Unlike recent methods that fundamentally alter the generation process in DDMs, our GENIE solves the true generative DE and still enables applications such as encoding and guided sampling. Project page and code: https://nv-tlabs.github.io/GENIE.
BibTeX
@inproceedings{dockhorn2022genie,
  author = {Tim Dockhorn and Arash Vahdat and Karsten Kreis},
  title = {{GENIE: Higher-Order Denoising Diffusion Solvers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=LKEYuYNOqx},
}
Figure