Automated Derivation of Probabilistic Inference

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2106 papers116 conceptsfrozen 2026-10-04

For decades, applying Bayesian inference to a new problem required what Mohammad Emtiyaz Khan calls "math gymnastics." If a researcher wanted to move from a simple Gaussian model to one with a Student-t distribution or add a latent hierarchy, they had to manually re-derive complex update equations or gradients using calculus and expectations. This manual bottleneck meant that the speed of research was limited by the researcher's ability to solve integrals, rather than the quality of their ideas.

A massive research effort has shifted this paradigm from manual derivation to automated synthesis. By treating probabilistic models as programs and leveraging automatic differentiation, the field has moved toward "black-box" inference engines where the math is derived by the machine, not the human.

The Manual Bottleneck

In the early days of graphical models, as outlined in the foundational tutorial by Michael I. Jordan, Zoubin Ghahramani, and colleagues in An Introduction to Variational Methods for Graphical Models, variational inference (VI) was a bespoke process. To approximate a posterior distribution, one had to find a lower bound on the log-evidence (the ELBO) and derive coordinate-ascent updates specific to that model's structure.

This created a high barrier to entry. As Christopher M. Bishop and John M. Winn noted in their work on VIBES, even minor modifications to a model—like changing a hyperparameter's scope—necessitated a complete re-derivation of the inference math. This tight coupling between model structure and mathematical implementation discouraged exploratory research.

Compiling Models into Code

The first step toward automation was symbolic. Systems like AUTOBAYES, developed by Alexander G. Gray and Bernd Fischer, acted as "statistical algorithm compilers." They used symbolic program synthesis to transform high-level model specifications into optimized C++ or Matlab code Automatic Derivation of Statistical Algorithms: The EM Family and Beyond.

This evolved into the field of Probabilistic Programming Languages (PPLs). Languages like Church, introduced by Noah D. Goodman and Joshua B. Tenenbaum, allowed researchers to define models as generative procedures Church: a language for generative models. The key insight was that if a model is a program, inference can be treated as a nonstandard interpretation of that program Nonstandard Interpretations of Probabilistic Programs for Efficient Inference. Instead of executing the code to generate data, the system analyzes the code to compute gradients or track dependencies.

Black-Box Variational Inference

The true "push-button" revolution arrived with Automatic Differentiation Variational Inference (ADVI). Developed by Alp Kucukelbir, Rajesh Ranganath, and Andrew Gelman, ADVI automates the derivation of variational algorithms by combining three ideas:

  1. Coordinate Transformation: Automatically mapping constrained parameters (like a positive variance) to an unconstrained real space.
  2. The Reparameterization Trick: Expressing random variables as deterministic functions of a noise source, allowing gradients to flow through expectations.
  3. Automatic Differentiation: Using the same technology that powers deep learning to calculate the gradients of the ELBO without manual derivation Automatic Differentiation Variational Inference.

This allowed frameworks like Stan, Pyro, and Edward to provide a unified toolchain where the user only provides the model and the data Deep Probabilistic Programming Languages: A Qualitative Study.

Structural Automation and Hybrid Inference

While ADVI handles the calculus, other researchers focused on automating the structure of the inference. A common technique to improve sampling is Rao-Blackwellization, which involves analytically integrating out (marginalizing) some variables to reduce the variance of the estimator.

Historically, this required expert statisticians to identify tractable sub-components. However, Lawrence M. Murray and Daniel Lundén introduced Delayed Sampling, a runtime mechanism that automatically discovers and solves these tractable parts of a model using a dynamic relationship graph Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. Similarly, Waïss Azizian and Guillaume Baudart demonstrated how to automatically combine Sequential Monte Carlo with exact Belief Propagation Automatic Rao-Blackwellization for Sequential Monte Carlo with Belief Propagation.

The Neural Integration

The automation of inference has also enabled the rise of Neural Topic Models. Traditional models like Latent Dirichlet Allocation (LDA) required complex, model-specific derivations for every variation. By applying autoencoding variational inference, Akash Srivastava and Charles Sutton showed that one could replace these derivations with a neural "inference network" Autoencoding Variational Inference For Topic Models. This modularity allows researchers to swap out components—like using a GAN-based discriminator or a Dirichlet prior—without re-engineering the entire inference pipeline ATM: Adversarial-neural Topic Model.

By decoupling the model specification from the inference implementation, these automated systems have turned Bayesian modeling from a mathematical marathon into an iterative engineering discipline.

Papers in this version

Showing 60 of 106 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 72 papers
  2. Version 1 · 2026-10-04 · 106 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manual_derivation_overhead_467fa480,
  title = {manual derivation overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-derivation-overhead/rdr_467fa480001e42deaf4a334b90b2ef57},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_derivation_of_probabilistic_in_467fa480_v1,
  title = {Automated Derivation of Probabilistic Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-derivation-of-probabilistic-inference/rdr_467fa480001e42deaf4a334b90b2ef57/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}