Most modern generative models, from Stable Diffusion to large-scale energy-based models, rely on the score function—the gradient of the log-density of the data. This first-order information acts like a compass, pointing a sampling algorithm toward regions of higher probability. However, a compass only tells you which direction to face; it doesn't tell you how the terrain is curving or how far you can safely step before the path changes.
A growing body of research suggests that by incorporating second-order information—specifically the Hessian of the log-density—we can transform these "compass-based" samplers into "map-based" navigators. By accounting for the local curvature of the data manifold, these methods achieve faster convergence, higher sample fidelity, and better handling of complex constraints.
From Gradients to Curvature
In standard score-based diffusion, the reverse process is essentially a first-order discretization of a stochastic differential equation. Because these models only know the local gradient, they must take very small steps to avoid overshooting or diverging, especially in high-dimensional spaces where the "probability ridges" are narrow and sharp.
Yuchen Liang and colleagues demonstrate in Broadening Target Distributions for Accelerated Diffusion Models via a Novel Analysis Approach that integrating the Hessian allows the sampler to track the reverse trajectory much more precisely. Intuitively, while the score tells you the slope, the Hessian tells you the "bowl-likeness" of the local area. This allows the algorithm to cancel out dominant discretization errors that typically plague first-order methods.
This theoretical leap is quantified by Gen Li and Yuting Wei in Accelerating Convergence of Score-Based Diffusion Models, Provably. They show that second-order approximations can improve the convergence rate of deterministic samplers from $O(1/T)$ to $O(1/T^2)$. In practical terms, this means achieving the same image quality with significantly fewer sequential function evaluations.
Navigating Complex Manifolds
Second-order information is particularly transformative when the sampling task involves constraints or complex geometries. In Geometry-Correct Diffusion Posterior Sampling with Denoiser-Pullback Curvature Guidance and Manifold-Aligned Damping, Seunghyeok Shin and his team introduce CLAMP, which uses "curvature-aware" corrections to solve inverse problems (like medical imaging reconstruction). Instead of using a fixed step size, the algorithm uses the Hessian to "pull" the measurement information through the denoiser, adapting the update to the local shape of the data manifold.
Similarly, Marc Anton Finzi and Andrew Gordon Wilson show in User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems that the Jacobian of the score function—which is the Hessian of the log-probability—provides a precise directional map for moving toward rare "extreme events" in chaotic systems. This allows for sampling under physical constraints without the need to retrain the underlying model.
Learning the Hessian
A natural question arises: where does this second-order information come from? While some methods estimate it on the fly using Tweedie’s formula or automatic differentiation, others propose learning it directly.
- Likelihood Matching: Lei Qian and Song Xi Chen propose Likelihood Matching for Diffusion Models, a framework that optimizes the data likelihood by incorporating the Hessian into the training objective itself. This ensures the model learns not just the direction of the data, but its density's curvature.
- Missing Data: In MissScore: High-Order Score Estimation in the Presence of Missing Data, Wenqin Liu and Biwei Huang develop a way to learn high-order scores even when the training data is incomplete, bypassing the need for biased pre-imputation.
- Stein Variational Methods: Gianluca Detommaso and Tiangang Cui utilize the Stein Variational Newton (SVN) method in Stein Variational Online Changepoint Detection, transporting particles toward a target posterior by leveraging second-order Hessian information to improve convergence in high-dimensional parameter spaces like LSTMs.
Theoretical Guarantees and Dimension Independence
The most profound promise of second-order sampling is its potential to break the "curse of dimensionality." Standard Langevin dynamics often suffer from scaling penalties as the number of dimensions $d$ increases.
Sinho Chewi and Philippe Rigollet demonstrate in Exponential ergodicity of mirror-Langevin diffusions that by setting a "mirror map" to the potential of the target distribution—effectively a second-order optimization approach—one can achieve convergence rates that are independent of dimensionality.
Furthering this, Yifeng Yu and Lu Yu show in Advancing Wasserstein Convergence Analysis of Score-Based Models that a Hessian-informed "Second-Order" (SO) sampler achieves a convergence rate of $\widetilde{\mathcal{O}}(\sqrt{d}/\varepsilon)$ in Wasserstein distance. This is a massive improvement over the standard $\widetilde{\mathcal{O}}(d/\varepsilon^2)$ rate, proving that curvature is not just a heuristic, but a fundamental tool for efficient high-dimensional generative modeling.