Executive Summary
Autoconj is a system that automatically discovers and exploits "conjugacy" in probabilistic models written in plain Python and NumPy. By transforming standard code into a canonical mathematical form, it eliminates the need for researchers to manually derive complex conditional distributions or rely on restrictive, monolithic probabilistic programming languages (PPLs).
The Challenge of Conjugacy in Bayesian Modeling
In Bayesian modeling, "conjugacy" is a mathematical convenience where the posterior distribution belongs to the same family as the prior. When a model has this property, inference becomes much easier: researchers can perform Gibbs sampling, coordinate-ascent variational inference (CAVI), or marginalize out variables using simple, closed-form updates.
However, researchers typically face a frustrating trade-off when trying to leverage conjugacy:
- Manual Derivation: Deriving these updates by hand is time-consuming, mathematically taxing, and highly prone to bugs.
- Domain-Specific Languages (DSLs): Systems like BUGS or VIBES automate these derivations but act as "walled gardens." They are difficult to compose with modern Python tools (like deep learning frameworks) and force users to write models in specialized, limited syntax.
The authors address this bottleneck by asking: Can we achieve the automation of a DSL while staying entirely within the flexible Python/NumPy ecosystem?
How Autoconj Works
Autoconj operates by "looking under the hood" of a standard Python function that computes a log-joint distribution. It follows a three-step process:
- Tracing: Using a tracer borrowed from the Autograd library, Autoconj executes the Python function once with dummy arguments. It records every primitive operation (like additions, multiplications, or NumPy calls) to build a Directed Acyclic Graph (DAG) representing the computation.
- Graph Rewriting: This is the system's core innovation. A domain-specific rewrite engine manipulates the graph into a canonical form. It specifically attempts to represent the log-joint as a sum of
np.einsumoperations (tensor contractions). This mathematical normalization makes the underlying structure of the exponential family visible. - Algebraic Extraction: Once in canonical form, Autoconj does not simply look for "known pairs" of distributions in a lookup table. Instead, it uses symbolic differentiation to extract the sufficient statistics and natural parameters of the conditional distributions.
Because it operates on algebraic structure rather than a hard-coded list of distribution pairs, Autoconj is remarkably flexible.
Core Innovations
- Conjugacy as Algebra, Not Lookup: Most PPLs treat conjugacy as a lookup table (e.g., "If Prior is Beta and Likelihood is Binomial, then Posterior is Beta"). Autoconj treats it as a structural property of the log-joint function's graph. If the math resolves to a multilinear polynomial of sufficient statistics, Autoconj can exploit it—even if the user never explicitly named the distributions.
- Hermetic Rewriting: The system is "hermetic," meaning it takes a Python function, transforms it, and returns a new Python function. This allows it to be used seamlessly as a pre-processing step for other tools like SciPy optimizers or TensorFlow.
- Support for "Quasi-Conjugacy": Autoconj can also work with variational lower bounds (such as the Jaakkola-Jordan bound for logistic regression). This extends the benefits of conjugate-style updates to models that are not strictly conjugate.
Evaluation and Key Results
As a machine learning systems paper, Autoconj is evaluated using classic probabilistic models (Beta-Bernoulli, Normal-Gamma, Bayesian Logistic Regression, Factor Analysis, and Gaussian Mixture Models) to test exact conditioning, marginalization, and hardware acceleration.
- Ease of Use: Complex recursions, such as the Kalman filter, can be implemented in just a few lines of code by repeatedly calling Autoconj's
marginalize()andcomplete_conditional()functions. - Performance: By translating derived updates into TensorFlow, Autoconj can leverage GPU acceleration. In a GMM benchmark, 500 iterations of variational inference took 62.9 seconds on a CPU (NumPy) but only 4.3 seconds on a GPU (TensorFlow)—an order-of-magnitude speedup with zero manual derivation.
- Inference Quality: In Factor Analysis tests, Autoconj’s automated CAVI converged significantly faster than standard Gibbs sampling and provided much tighter fits than "black-box" methods like Automatic Differentiation Variational Inference (ADVI).
Limitations and Future Work
- Control Flow: The tracing mechanism only follows a single execution path. If a model contains
if/elsestatements that depend on the values of random variables, the tracer will fail to capture the full logic. - Support for Non-Exponential Families: The system is fundamentally built around the properties of exponential families. Models that fall outside this scope (e.g., heavy-tailed distributions like Cauchy) may not be handled as elegantly.
- Scaling Rewrites: While the rewrite system successfully handled the paper's examples, the authors note they lack a formal proof that the rewrites will always terminate or reach the same result (confluence) for every possible complex graph.
Summary updated