Abstract
Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and score matching objectives. To fill up this gap, we show that the negative likelihood of the ODE can be bounded by controlling the first, second, and third-order score matching errors; and we further present a novel high-order denoising score matching method to enable maximum likelihood training of score-based diffusion ODEs. Our algorithm guarantees that the higher-order matching error is bounded by the training error and the lower-order errors. We empirically observe that by high-order score matching, score-based diffusion ODEs achieve better likelihood on both synthetic data and CIFAR-10, while retaining the high generation quality.
Introduction
Score-based generative models (SGMs) have become a cornerstone of modern AI, powering tools like DALL-E and Stable Diffusion. These models work by learning the "score function"—the gradient of the log-probability density of the data. Once trained, data can be generated using either a Stochastic Differential Equation (SDE) or a deterministic Ordinary Differential Equation (ODE), known as a "Probability Flow ODE" or ScoreODE.
The ScoreODE is particularly attractive because it allows for exact likelihood evaluation and faster sampling. However, standard training (first-order score matching) is mathematically equivalent to maximizing the likelihood of the SDE, but it doesn't necessarily maximize the likelihood of the ODE. As shown in Figure 1, a model trained only with first-order score matching can fail to accurately represent the data density in deterministic mode, leading to poor likelihood scores.
Methodology
The authors bridge this gap by systematically analyzing the KL divergence of ScoreODEs. They prove that the ODE's likelihood objective contains extra terms involving the curvature of the probability flow that first-order score matching ignores. To address this, they show that the negative log-likelihood of the ODE can be upper-bounded by controlling the errors of the first, second, and third-order score functions.
To minimize these bounds, the researchers developed a novel High-Order Denoising Score Matching (DSM) objective. Unlike previous high-order methods that were often numerically unstable, this paper proposes an "error-bounded" version. By using stop-gradient operations on lower-order terms, they ensure that the higher-order matching error is strictly controlled by the training loss and lower-order errors. To make this computationally feasible for high-dimensional images, they employ the Hutchinson trace estimator and efficient "Jacobian-vector-product" computations in JAX, avoiding the prohibitive $O(d^2)$ cost of computing full Hessian matrices.
Key Results
The proposed high-order DSM consistently improved the likelihood of ScoreODEs across both synthetic and real-world benchmarks:
- CIFAR-10 (VE type): The negative log-likelihood (NLL) improved from 3.66 bpd (standard) to 3.38 bpd (third-order), a significant jump in density modeling performance.
- ImageNet 32x32: NLL improved from 4.21 bpd to 4.04 bpd.
- Qualitative Density: On 2-D checkerboard tasks, the third-order method produced much smoother and more accurate density maps compared to the noisy, "streaky" density produced by first-order models.
- Sample Quality: Importantly, training for better ODE likelihood did not degrade visual quality. FID scores remained competitive (e.g., 2.42 vs 2.95 on CIFAR-10), confirming that likelihood gains are compatible with high-quality generation.
Conclusion
This work demonstrates that first-order score matching is insufficient for optimal deterministic diffusion. By incorporating second- and third-order derivatives into the training objective, the authors provide a more robust framework for maximum likelihood training of ScoreODEs. While high-order matching introduces a computational overhead (roughly 1.5x to 2x training time), the resulting improvements in density estimation and likelihood evaluation make it a valuable tool for the next generation of score-based models.
Summary updated