Compositional Score Modeling for Scalable Amortized Inference

This direction is merged into another direction: likelihood-free inference (intractable likelihood functions). Its last version stays readable and citable here.

Version 1 of 110 papers20 conceptsfrozen 2026-10-04

Simulation-Based Inference (SBI) allows researchers to perform Bayesian inference even when the likelihood function of a model is "intractable"—meaning we can simulate data from the model, but we cannot easily write down its mathematical form. While traditional Amortized Bayesian Inference (ABI) is fast at test time, it struggles to scale. If you train a model on datasets of size $N=10$, it typically cannot generalize to $N=100$ without retraining. Furthermore, simulating massive, nested datasets for hierarchical models is often computationally prohibitive.

A new direction in the field, led by researchers like Tomas Geffner, George Papamakarios, and Jonas Arruda, addresses this by shifting from predicting distributions to predicting scores—the gradients of the log-posterior density.

The Additive Property of Scores

The core intuition behind this research is a simple property of logarithms. In Bayesian inference, the joint posterior for a set of independent observations $x_{1:n}$ is proportional to the prior times the product of individual likelihoods: $$p(\theta | x_{1:n}) \propto p(\theta) \prod_{i=1}^n p(x_i | \theta)$$

When we take the gradient of the log of this expression (the "score"), the product becomes a sum: $$\nabla_\theta \log p(\theta | x_{1:n}) = \nabla_\theta \log p(\theta) + \sum_{i=1}^n \nabla_\theta \log p(x_i | \theta)$$

As Tomas Geffner and colleagues demonstrate in Compositional Score Modeling for Simulation-Based Inference, this "factorization" is incredibly powerful. Instead of training a massive neural network to understand a dataset of size $N$, we can train a model to estimate the score of a single observation. At inference time, we simply sum these individual score estimates to form the total posterior score. This allows for "plug-and-play" inference with any number of observations without retraining.

The Challenge of Compounding Errors

While the math is elegant, the implementation faces a significant hurdle: error compounding. Every neural network estimate of a score contains a small amount of approximation error. When you sum these estimates across thousands of observations or hierarchical groups, these tiny errors aggregate.

In Diffusion Models in Simulation-Based Inference: A Tutorial Review, Stefan T. Radev and Jonas Arruda note that these compounded errors often lead to unstable sampling trajectories or "over-confident" posteriors that don't accurately reflect true uncertainty. This is particularly problematic in the high-noise phases of the diffusion process, where the model is far from the data manifold and the gradients are most prone to numerical instability.

Stabilizing Inference via Score Damping

To solve this, Jonas Arruda and Jan Hasenauer introduced a "divide-and-conquer" strategy for large-scale hierarchical models in Compositional amortized inference for large-scale hierarchical Bayesian models. Their key innovation is time-dependent score damping.

The mechanism works by suppressing the influence of individual score terms during the early, high-noise stages of the reverse diffusion process. As the sampling trajectory approaches the data manifold (low noise), the damping factor is relaxed. This prevents the initial "random walk" from diverging due to accumulated estimation errors, allowing the model to scale to hundreds of thousands of parameters—tasks that previously required days of MCMC computation but can now be completed in minutes.

Why the Error Doesn't Explode

A natural question arises: if we keep adding observations, won't the error eventually become unmanageable? Theoretical work by Camille Touron and Julyan Arbel in Error analysis of a compositional score-based algorithm for simulation-based inference suggests a surprising stability.

They found that while individual errors do accumulate, the precision (the inverse of the variance) of the combined posterior also increases with more data. This increasing precision introduces a reciprocal term that counteracts the linear growth of the sum of errors. In essence, as the posterior becomes sharper and more informative, it becomes more robust to the noise in the individual score estimates. This explains why compositional methods can scale to large catalogs of events, such as those seen in gravitational wave astronomy K. Leyde et al., without the accuracy degrading as the dataset grows.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_score_modeling_for_scalabl_ecdc7ae3,
  title = {Compositional Score Modeling for Scalable Amortized Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-score-modeling-for-scalable-amortized-inference/rdr_ecdc7ae32e154a87b2879e0518b48b99},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_score_modeling_for_scalabl_ecdc7ae3_v1,
  title = {Compositional Score Modeling for Scalable Amortized Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-score-modeling-for-scalable-amortized-inference/rdr_ecdc7ae32e154a87b2879e0518b48b99/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}