Automatic structured variational inference

Luca Ambrogioni, Max Hinne, Marcel van Gerven

Abstract

Probabilistic Programming Languages (PPLs) simplify Bayesian inference by automating the construction of inference engines. However, Stochastic Variational Inference (SVI) often relies on the mean-field approximation, which ignores crucial dependencies between latent variables. We introduce Automatic Structured Variational Inference (ASVI), a fully automated method for constructing structured variational families. ASVI mimics the closed-form updates of conjugate Bayesian models by blending prior parameters with learnable parameters via a convex-update rule. This allows the variational family to capture dependencies present in the model's forward pass with $O(N)$ complexity. We demonstrate that ASVI achieves state-of-the-art performance on tasks ranging from low-dimensional time-series to high-dimensional image smoothing.

Introduction

The power of Probabilistic Programming lies in decoupling model specification from inference. While SVI provides a scalable optimization framework for inference, its performance is heavily dependent on the choice of the variational family. The standard automated choice, the mean-field approximation, assumes independence between all latent variables, often leading to poor uncertainty estimates. Conversely, structured families that capture these dependencies typically require manual derivation and model-specific implementation.

Existing automated structured methods often face scalability issues. For instance, full-covariance Gaussians scale quadratically with the number of parameters. Other methods may ignore the model's prior structure or require strict mathematical conditions like local conjugacy. ASVI addresses these challenges by providing an automated, scalable, and structured variational family that respects the original model's architecture.

The ASVI Framework

ASVI introduces convex-update variational families. Rather than mapping variables to an unconstrained space, ASVI maintains the same distribution families and control flow as the original probabilistic program.

Convex Update Rule

In conjugate Bayesian models, the exact posterior parameters are often a convex combination of prior parameters and data. ASVI adopts this principle. For every variable in the model, the variational posterior uses the same distribution family as the prior. The parameters are calculated as a convex combination: $\lambda \odot \theta_{prior} + (1-\lambda) \odot \alpha$, where $\theta_{prior}$ is derived from the model's forward pass, $\alpha$ is a learnable parameter, and $\lambda \in [0, 1]$ is a learnable gate that balances the prior influence against the new parameter.

Implementation and Complexity

By intercepting the forward pass using effect handlers, the variational program inherits the exact dependency graph and control flow of the original model. This ensures that ASVI maintains $O(N)$ time and space complexity, making it as efficient as the mean-field approach while being significantly more expressive.

Experimental Evaluation

We evaluated ASVI across several benchmarks, including the Inference Gym suite and high-dimensional image tasks.

Time-Series and Hierarchical Models

On tasks like the stochastic Lorenz system, ASVI demonstrates a superior ability to track ground truth trajectories compared to mean-field baselines.

ASVI tracks ground truth in the Lorenz system more accurately than Mean Field or Large IAF baselines.

In the Inference Gym, ASVI consistently outperformed Mean Field and Multivariate Normal baselines. Notably, it matched the performance of a large Inverse Autoregressive Flow (IAF) while using significantly fewer parameters and requiring fewer iterations to converge.

High-Dimensional Image Smoothing

ASVI was also tested on deep Bayesian smoothing tasks using CIFAR10 and FashionGEN datasets. In these tasks, the goal is to reconstruct image trajectories from noisy observations.

Qualitative comparison of image reconstructions. ASVI produces sharper, more accurate reconstructions than the Mean Field baseline.

ASVI yielded tighter Evidence Lower Bounds (ELBOs) and produced visibly sharper image reconstructions than mean-field and linear coupling baselines. In amortized generative modeling (VAEs) on MNIST, ASVI achieved a test ELBO of -134.6, a substantial improvement over the -153.8 achieved by the mean-field baseline.

Discussion and Future Work

ASVI offers several key advantages, including prior consistency (reverting to the prior when $\lambda = 1$), extreme parsimony ($2P$ parameters), and the ability to handle discrete variables and stochastic control flow naturally.

The main limitation of ASVI is its inability to capture "explaining away" dependencies—correlations induced by observations in v-structures. In models with decoupled prior weights, such as standard Bayesian Neural Networks, ASVI reverts to a mean-field approximation. Future work will focus on extending these structured surrogates to capture such posterior dependencies while maintaining automation and scalability.

Summary updated

BibTeX
@inproceedings{ambrogioni2021automatic,
  author = {Luca Ambrogioni and Max Hinne and Marcel van Gerven},
  title = {{Automatic structured variational inference}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
}
Figure