Score Based Inference for Unnormalized Models

157 concepts104 papers2 proposals

In probabilistic modeling, we frequently encounter distributions that are easy to specify up to a constant factor, but incredibly difficult to normalize. We write these as:

$$p(x) = \frac{\tilde{p}(x)}{Z}$$

where $\tilde{p}(x)$ is the unnormalized density (or energy function) that we can easily compute, and $Z = \int \tilde{p}(x) dx$ is the normalization constant, or partition function.

Computing $Z$ requires integrating over the entire data space. In high dimensions, this integral quickly becomes intractable, creating a massive bottleneck for maximum likelihood estimation, hypothesis testing, and Bayesian inference.

The core intuition behind score-based inference is simple yet transformative: if we take the derivative of the log-density with respect to the input variable $x$, the troublesome partition function completely vanishes. The resulting vector field is called the score function:

$$\psi(x) = \nabla_x \log p(x) = \nabla_x \log \tilde{p}(x) - \nabla_x \log Z = \nabla_x \log \tilde{p}(x)$$

Because $\nabla_x \log Z = 0$, the score function depends entirely on the unnormalized model $\tilde{p}(x)$. By shifting our focus from estimating absolute probabilities to estimating the gradients of log-probabilities, we can completely bypass the partition function.


Learning Densities via Score Matching

Instead of maximizing the likelihood of the data (which requires knowing $Z$), we can train a model by matching its score field to the score field of the true data distribution. This paradigm, known as Score Matching, minimizes the Fisher divergence between the data distribution $p_{data}$ and the model $p_{model}$:

$$\mathcal{J} = \mathbb{E}_{p_{data}} \left[ \frac{1}{2} \|\nabla_x \log p_{model}(x) - \nabla_x \log p_{data}(x)\|^2 \right]$$

Since the true data score $\nabla_x \log p_{data}(x)$ is unknown, we can use integration by parts to rewrite this objective so that it only depends on the model's derivatives and the observed samples. As Swersky et al. demonstrate in their work on On Autoencoders and Score Matching for Energy Based Models, this formulation provides a direct bridge for optimizing energy-based models without partition functions.

To scale this to complex, infinite-dimensional settings, Sutherland et al. developed scalable estimators using Nyström kernel approximations in Efficient and principled score estimation with Nyström kernel exponential families. Furthermore, Pabbaraju et al. proved in Provable benefits of score matching that for certain exponential families, score matching is exponentially faster to compute than maximum likelihood estimation, turning an NP-hard optimization problem into a computationally efficient alternative.

The flexibility of score matching has allowed researchers to move beyond standard Euclidean spaces:


Goodness-of-Fit Testing and Stein's Method

How do we evaluate whether a set of empirical samples actually fits an unnormalized target distribution? Traditional statistical tests rely on evaluating the density function directly, which fails when $Z$ is unknown.

To solve this, researchers combine score functions with Stein's Method. A Stein operator $\mathcal{A}_p$ is constructed such that for a wide class of test functions $f$, the expected value of the operator applied to $f$ is exactly zero if and only if the samples are drawn from the target distribution $p$:

$$\mathbb{E}_{x \sim p} [\mathcal{A}_p f(x)] = 0$$

Crucially, this operator relies entirely on the score function $\nabla_x \log p(x)$. In Measuring Sample Quality with Stein's Method, Gorham et al. leveraged this property to bound asymptotic bias and evaluate sample quality without partition functions.

  Empirical Samples (X) ---------> [ Stein Operator A_p ] <--------- Target Score ∇_x log p(x)
                                             |
                                             v
                                  [ Reproducing Kernel k ]
                                             |
                                             v
                              Kernelized Stein Discrepancy (KSD)
                               (Zero if samples perfectly match)

By pairing this operator with reproducing kernel Hilbert spaces, Chwialkowski et al. (A Kernel Test of Goodness of Fit) and Lee (A Kernelized Stein Discrepancy for Goodness-of-fit Tests) independently introduced the Kernelized Stein Discrepancy (KSD). KSD provides a non-parametric metric to test model fit using only the score function. Kanagawa later extended this normalization-invariant testing strategy to conditional density models in Testing Goodness of Fit of Conditional Density Models with Kernels.

Streaming and Sequential Tests

This framework is highly effective for streaming data applications where batch testing is impractical. Zhou et al. (Sequential Kernel Goodness-of-fit Testing) and Martinez-Taboada et al. (Sequential Kernelized Stein Discrepancy) integrated KSD into sequential testing frameworks, allowing researchers to monitor data streams and stop as soon as a distribution shift is detected.

Similarly, Wu et al. (Robust Quickest Change Detection for Unnormalized Models) and Diao et al. (Quickest Change Detection for Unnormalized Statistical Models) utilized the Hyvärinen score to perform real-time change detection in high-dimensional unnormalized data streams.


Expanding the Score-Based Toolkit

Beyond density estimation and hypothesis testing, score-based methods have transformed optimization and integration across computational statistics:

1. Variational Inference

Traditional variational inference optimizes the Evidence Lower Bound (ELBO), which requires global integration. In Variational Inference with Gaussian Score Matching, Modi et al. introduced an alternative that matches scores locally at sampled points. By focusing on the local geometry rather than a global objective, this approach requires drastically fewer gradient evaluations. To scale this to high-dimensional latent spaces, Modi et al. later introduced a low-rank covariance structure in Batch, match, and patch: low-rank approximations for score-based variational inference, reducing computational complexity from quadratic to linear.

2. Numerical Integration

Calculating expectations over unnormalized posteriors is a fundamental challenge in Bayesian statistics. By leveraging the Stein identity, numerical integration can be performed directly against unnormalized distributions. Ott et al. introduced Bayesian Stein Networks (BSNs) in Bayesian Numerical Integration with Neural Networks, demonstrating that neural networks equipped with a specialized Stein layer can perform high-accuracy integration using only the target distribution's score function.

3. Generative Modeling and Control Variates

By treating generation as a process of moving random points along a learned score field toward high-density regions via Langevin dynamics, Yang et al. demonstrated state-of-the-art 3D shape generation in Learning Gradient Fields for Shape Generation. Additionally, score functions serve as powerful variance-reduction tools; Li et al. (Multilevel Control Functional) showed that integrating Stein operators into multi-fidelity frameworks drastically reduces the variance of Monte Carlo estimators for complex, unnormalized densities.

Key Foundational Work

Estimation of Non-Normalized Statistical Models by Score Matching (2005) — contributed 6 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_score_matching_18b33f3d,
  title = {score matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/score-matching/rdr_18b33f3dd836449684b4e6e587539e44},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_score_matching_18b33f3d_v1,
  title = {score matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/score-matching/rdr_18b33f3dd836449684b4e6e587539e44/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake