Composing Non-Conjugate Factor Graphs with Closed-Form Variational Inference

Mykola Lukashchuk, Kyrylo Yemets, Wouter M. Kouw, Dmitry Bagaev, İsmail Şenöz, Jeff Beck, Bert de VriesarXiv 2026 · arXiv

TL;DR

Stacking probabilistic models into deep architectures usually breaks closed-form variational inference, forcing researchers to rely on computationally heavy or poorly calibrated black-box approximations. This paper demonstrates that closed-form inference can be completely preserved across deeply nested models by using a carefully selected "alphabet" of five factor-graph primitives. This framework achieves universal function approximation while naturally providing mathematically rigorous, calibrated uncertainty.

What Problem Does This Paper Address?

In traditional, deterministic programming, building complex systems is straightforward: we nest functions within functions, and complexity grows without any fundamental barriers. Probabilistic modeling desperately needs this same kind of compositional hierarchy to be truly expressive—for instance, to make a forecasting expert's reliability an input-dependent function that carries its own uncertainty.

However, stacking probabilistic building blocks into deeper architectures typically introduces non-conjugacy, which breaks closed-form inference. To circumvent this, the machine learning field has relied on black-box inference methods (like MCMC, black-box variational inference, or normalizing flows). While powerful, these approaches sacrifice closed-form tractability, can be computationally expensive, and often struggle with uncertainty calibration. This paper tackles a fundamental question: Can probabilistic models compose hierarchically while retaining exact, closed-form inference updates?

Methods

The authors address this challenge by designing a probabilistic framework structured like a programming language, complete with an alphabet, a grammar, and a runtime execution engine:

  • The Alphabet: The authors identify a five-letter alphabet of factor-graph primitives: a bilinear factor (softdot), an exponential link, a Gamma prior, a Gaussian likelihood, and an equality node.
Figure 1: The building blocks as factor graph nodes. Square nodes are factors; round nodes represent neighboring nodes to which these factors connect (e.g., priors, likelihoods, or other factors sharing the same variable). Filled black squares denote clamped variables: fixed numerical values that cannot be connected to other factors (e.g., the shape α of the gamma factor). This clamped notation is used throughout for any factor edge that carries a fixed constant rather than a latent. Orientation markers on the softdot factor (filled semi-circle on the w side, open on the ϕ side) allow edge roles to be identified even when the node is rotated in composed graphs. Edge styles distinguish variable roles: solid for weights, features, and observations (w, ϕ, y, µ), dashed for precisions (τ, γ, β), and dash-dotted for the latent z. The equality node enforces that all connected edges carry the same variable. Composing models amounts to connecting factors via shared variable edges of the same line type.
  • The Grammar: This defines how the primitives are composed into increasingly deep structures. The paper demonstrates three levels of depth: a static ensemble (Depth 0), input-dependent gating (Depth 1, termed Precision-Gated Experts), and split-branch routing (Depth 2). By stacking these routing layers, the grammar can encode arbitrary binary decision trees.
  • The Runtime: The inference engine minimizes the Bethe free energy using a combination of Variational Message Passing (VMP) and Belief Propagation (BP). By imposing mean-field factorization and exponential-family form constraints, messages passed along the graph edges retain known parametric forms (Gaussian or Gamma). The traditionally intractable non-conjugate bottleneck—the exponential link—is solved analytically using the Gaussian moment-generating function and the sufficient statistics of the Gamma family.

Datasets

The framework was evaluated on five standard multivariate time-series forecasting benchmarks:

  • ETTh1 (Electricity Transformer Temperature)
  • ETTh2
  • Exchange Rate
  • Electricity
  • Traffic

Key Insights and Ideas

  • A Message Type System: The line styles in the factor graphs (solid/dash-dot for Gaussian, dashed for Gamma) act like a type system in a programming language. Because each primitive preserves these message families, message computation depends purely on local factor properties, ensuring closed-form updates regardless of graph depth.
  • Universal Approximation via Exact Updates: The authors prove that stacking these routing layers can approximate any continuous function to arbitrary accuracy in the sharp-routing limit. This bridges the gap between the high expressivity of deep neural networks/decision trees and the mathematical tractability of classical conjugate models.
  • Inferred Gating Over Learned Gating: Instead of using standard neural network softmax layers to learn gating weights, this framework infers gating functions probabilistically. This prevents the model from becoming overconfident and provides well-calibrated uncertainty over expert selection.

Key Results

The framework was deployed as a Bayesian Mixture of Experts (MoE) for time-series forecasting, comparing various configurations against standard softmax-based MoE baselines:

  • The "Noisy Diagonal" Variant Wins: This configuration, which separates expert disagreement from irreducible predictive noise and applies a diagonal covariance constraint, provided the best aggregate trade-off between accuracy, calibration, and parameter efficiency.
  • Massive Parameter Savings: On the Exchange Rate dataset, the Noisy Diagonal model achieved the best Negative Log-Likelihood (NLL) by a wide margin while reducing the parameter count from 15,512 (in the full covariance version) down to just 952.
  • Superior Calibration Stability: Traditional softmax-based MoE baselines proved unstable, frequently yielding non-finite NLL values due to severe overconfidence and model collapse. In contrast, the proposed factor-graph framework maintained stable, calibrated predictive distributions across all horizons.

Limitations and Future Work

  • Iterative Fixed-Point Solutions: Although the update equations are derived in closed form, they are inherently non-linear. This requires solving them iteratively using natural gradient descent on a Riemannian manifold, rather than computing them in a single step.
  • Restricted Building Blocks: Modelers are constrained to using the five specified alphabet primitives, trading away the freedom of arbitrary differentiable operations for guaranteed closed-form inference.
  • Alphabet Minimality: The authors conjecture that at least one non-conjugate factor (the exponential link) is strictly necessary to achieve universal approximation, but establishing a formal lower bound on the minimum required alphabet size remains an open problem.

Summary updated

Abstract

Stacking probabilistic building blocks into deeper architectures typically breaks closed-form inference. We show that closed-form inference can be preserved. We identify five factor-graph primitives: a bilinear factor, an exponential link, a Gamma prior, a Gaussian likelihood, and an equality node, and prove that any model composed from them admits closed-form variational message passing. The construction works because each primitive preserves a small set of message families: under mean-field factorization, messages on Gaussian variables remain Gaussian and messages on precision variables remain Gamma, while the only non-conjugate interface, the exponential link, remains tractable through the Gaussian moment-generating function and the sufficient statistics of the Gamma family. We demonstrate composition at increasing depth, from static ensembles through input-dependent gating to split-branch routing, and show that stacking routing layers encodes arbitrary decision trees, establishing universal function approximation with closed-form inference. Applied to ensemble time-series forecasting, the framework yields a Bayesian mixture of experts in which gating functions are inferred rather than learned, providing calibrated uncertainty over expert selection across five benchmark datasets.
BibTeX
@misc{lukashchuk2026composing,
  author = {Mykola Lukashchuk and Kyrylo Yemets and Wouter M. Kouw and Dmitry Bagaev and İsmail Şenöz and Jeff Beck and Bert de Vries},
  title = {{Composing Non-Conjugate Factor Graphs with Closed-Form Variational Inference}},
  year = {2026},
  eprint = {2605.29467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.29467},
}
Figure