Abstract: Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds. Modern approaches employ amortized variational inference, which uses a neural network to predict the parameters of the approximate posterior. This paper introduces Amortized Conjugate Posterior (ACP), a hybrid method that combines the mathematical structure of classical variational bounds with the scalability of amortized inference to train Noisy-OR models effectively, even with limited data.
Introduction: The Challenge of Noisy-OR Models
The Noisy-OR model is a fundamental tool in probabilistic graphical modeling, particularly for representing "many-to-one" causal relationships. In this framework, multiple independent latent causes can contribute to a single observed effect. For instance, in medical diagnosis, various diseases might independently cause the same symptom.
Despite its utility, training Noisy-OR models is computationally demanding. The primary bottleneck is the intractability of the posterior distribution—the probability of latent causes given observed effects. Traditionally, researchers have faced a trade-off between two paradigms:
- Conjugate Dual Inference (CDI): This classical approach uses structured approximations. While theoretically sound, it often lacks scalability and can be overly rigid.
- Amortized Variational Inference (AVI): Modern AVI uses neural networks (encoders) to predict posterior parameters. While flexible and scalable, these "black-box" models often ignore the underlying generative structure, leading to poor performance when training data is scarce.
The Amortized Conjugate Posterior (ACP) Method
The authors propose Amortized Conjugate Posterior (ACP), a hybrid approach that leverages the "best of both worlds." Instead of a generic neural network, ACP uses a posterior form derived from the model's mathematical structure.
Structured Amortization
ACP defines the "shape" of the approximate posterior using a factorized Bernoulli distribution, where parameters are tied to the model's generative weights ($\theta$). To achieve scalability, a neural network is trained to predict these variational parameters ($\psi$) directly from the input data. This amortization allows for fast inference without sacrificing the structural integrity of the model.
Optimization Objective
Unlike classical methods that focus on tightening mathematical bounds, ACP directly optimizes the Evidence Lower Bound (ELBO). To handle the discrete nature of the binary latent variables, the authors employ the Gumbel-Softmax reparameterization trick, enabling gradient-based optimization through the stochastic nodes.
Experimental Evaluation
The effectiveness of ACP was tested across several synthetic and real-world benchmarks:
- SYN-PATTERN & MULTI-MNIST: These datasets involve recovering known latent patterns (8x8 grids or MNIST digits).
- SYN-RANDOM: A dataset with varying sparsity levels to test robustness against different causal structures.
- NeurIPS Titles: A topic modeling task using paper titles to evaluate real-world applicability.
Key Results and Insights
The experiments revealed that incorporating model structure acts as a powerful regularizer. ACP demonstrated significant improvements in both data efficiency and recovery accuracy.
Superior Pattern Recovery
In the SYN-PATTERN task with a limited training set of only 20 examples, ACP achieved an F1 score of 76.1, vastly outperforming the 49.2 achieved by standard AVI. As shown in the qualitative results, ACP successfully recovers the underlying geometric structures that AVI fails to capture.
Generalization and Topic Coherence
On the SYN-RANDOM dataset, ACP consistently provided a better fit (lower Negative Log-Likelihood) than AVI, particularly in data-sparse regimes. In the NeurIPS topic modeling task, ACP produced more coherent word groupings, achieving a Point-wise Mutual Information (PMI) score of 2.75 compared to AVI's 2.55.
Conclusion
ACP represents a significant step forward in bridging the gap between classical structured inference and modern deep learning. By "baking in" the generative model's logic into the inference network, the authors achieve superior performance in low-data scenarios. While ACP currently requires manual derivation of the posterior form, it sets the stage for future research into automated discovery of structured variational distributions.
Summary updated