Inference and Learning for Unnormalized Statistical Models

This direction is merged into another direction: partition function intractability (partition function estimation). Its last version stays readable and citable here.

Version 1 of 163 papers63 conceptsfrozen 2026-10-04

In many areas of machine learning, we want to model data using a probability density function $p(x)$. A common way to define these models is through an "energy function" $E(x)$, where the probability is proportional to $e^{-E(x)}$. However, to make this a valid probability distribution that sums to one, we must divide by a normalization constant $Z$, also known as the partition function:

$$p(x; \theta) = \frac{1}{Z(\theta)} \tilde{p}(x; \theta)$$

As Simon Barthelmé and Nicolas Chopin point out in The Poisson transform for unnormalised statistical models, calculating $Z(\theta)$ is often a nightmare. It requires integrating the unnormalized density $\tilde{p}(x)$ over the entire space of possible data. In high dimensions or on complex manifolds, this integral is computationally intractable, effectively "locking" the model and preventing us from using standard tools like Maximum Likelihood Estimation (MLE).

A massive body of research has emerged to bypass this "partition function bottleneck," allowing us to train and evaluate models without ever calculating $Z$.

The Score Matching Intuition

The most elegant way to get rid of a constant is to take a derivative. If we look at the score function—the gradient of the log-density with respect to the data $x$—the normalization constant simply vanishes:

$$\nabla_x \log p(x; \theta) = \nabla_x \log \tilde{p}(x; \theta) - \nabla_x \log Z(\theta) = \nabla_x \log \tilde{p}(x; \theta)$$

Because $\log Z(\theta)$ does not depend on $x$, its derivative is zero. Aapo Hyvarinen pioneered this approach, showing that we can learn model parameters by minimizing the squared distance between the model's score and the data's score.

This idea has been extended to handle increasingly complex scenarios. For instance, Siwei Lyu demonstrated in Interpretation and Generalization of Score Matching that this technique is mathematically equivalent to minimizing the derivative of the KL divergence when data is perturbed by noise. More recently, researchers like Lina Lin and Ali Shojaie have applied Regularized Score Matching to estimate high-dimensional graphical models where traditional MLE would be impossible.

Stein Discrepancy: Testing Without Normalizing

While score matching helps us train models, we also need ways to test them. How do we know if a set of samples actually fits an unnormalized model?

Jason D. Lee, Qiang Liu, and Michael Jordan introduced Kernelized Stein Discrepancy (KSD) to solve this. KSD uses a "Stein operator" that, like score matching, relies only on the derivative of the log-density. By combining this operator with kernel methods, they created a goodness-of-fit test that works for any differentiable unnormalized density.

This framework is so robust that Alessandro Barp showed in Minimum Stein Discrepancy Estimators that many existing methods, including score matching and contrastive divergence, are actually specific instances of a broader "Stein" family.

Moving Beyond Flat Space

Real-world data often doesn't live on a flat Euclidean plane. It lives on spheres (directional data), tori (protein structures), or Lie groups (3D rotations). In these spaces, the normalization constant is even harder to compute because the integration happens over curved manifolds.

  • Rotations: Matthew Giamou and colleagues explored how to represent uncertainty over rotations using the Bingham distribution, noting that standard likelihood losses are often discarded for proxy objectives because $Z$ is too complex.
  • Lie Groups: Xiaoda Qu and Xiran Fan extended Stein methods to Lie groups, allowing for "normalization-free" inference on the geometric spaces used in robotics and computer vision.
  • Truncated Domains: When data is restricted to a specific region (like geographic data bounded by a border), the normalization constant changes based on the boundary. Abhin Shah and Devavrat Shah developed computationally efficient loss functions specifically for these truncated exponential families.

Is Bypassing $Z$ Statistically Sound?

A natural concern is whether we lose accuracy by ignoring the partition function. If MLE is the "gold standard" for efficiency (achieving the lowest possible variance), do these "shortcut" methods perform worse?

Song Liu and Takafumi Kanamori addressed this in Fisher Efficient Inference of Intractable Models. They introduced the Discriminative Likelihood Estimator (DLE), which uses Stein operators to achieve Fisher efficiency. This means that, theoretically, you can be just as accurate as if you had calculated the impossible normalization constant, provided you have enough data.

By shifting the focus from the value of the probability (which requires $Z$) to the shape of the probability (the score), this research direction has turned "intractable" models into practical tools for high-dimensional science and engineering.

Papers in this version

Showing 60 of 63 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inference_and_learning_for_unnormalized__6b86ad63,
  title = {Inference and Learning for Unnormalized Statistical Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inference-and-learning-for-unnormalized-statistical-models/rdr_6b86ad637f3f46c49d7fc5745092f99d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_inference_and_learning_for_unnormalized__6b86ad63_v1,
  title = {Inference and Learning for Unnormalized Statistical Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inference-and-learning-for-unnormalized-statistical-models/rdr_6b86ad637f3f46c49d7fc5745092f99d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}