Amortized Inference of Variational Bounds for Learning Noisy-OR

Yiming Yan, Melissa Ailem, Fei Sha

TL;DR

This paper introduces Amortized Conjugate Posterior (ACP), a novel hybrid approach to variational inference that combines the strengths of classical methods and modern amortized inference. ACP leverages classical techniques to derive structured approximate posteriors and then uses neural networks to learn the parameters of these posteriors, leading to improved performance, especially in low-data regimes.

What Problem Does This Paper Address?

Approximate inference in probabilistic graphical models is crucial for learning from data. Traditional methods rely on hand-crafted variational distributions and bounds that exploit model structure but can be complex to derive and may not scale well. On the other hand, modern Amortized Variational Inference (AVI) uses neural networks to learn posteriors directly, offering flexibility but often suffering from overfitting and high variance when training data is scarce. This paper addresses the challenge of achieving robust and efficient inference by bridging these two approaches, aiming to get the best of both worlds: the structural advantages of classical methods and the learning efficiency of neural networks.

Methods

The core contribution is the Amortized Conjugate Posterior (ACP) method. ACP takes a hybrid approach:

  • Classical Foundation: It uses classical techniques, specifically Conjugate Dual Inference (CDI), to derive specific, structured forms for the approximate posterior distribution. This leverages the known structure of the generative model (in this case, the NOISY-OR model) to create a more informed and potentially less complex posterior.
  • Amortized Learning: Instead of optimizing the variational parameters to achieve the "tightest bounds" as in traditional CDI, ACP uses a neural network (MLP) to learn these variational parameters. This "amortizes" the inference process, meaning the neural network learns to map input data points to the optimal variational parameters for their posteriors.
  • ELBO Maximization: The learned variational parameters are optimized to maximize the Evidence Lower Bound (ELBO), a standard objective in variational inference. This differs from traditional CDI, which focuses on optimizing bounds on the likelihood.

This hybrid approach allows the inference network to benefit from the model's structure while still being trained end-to-end to optimize the overall inference objective.

Datasets

The paper uses several synthetic datasets for evaluation:

  • SYN-PATTERN: Datasets with patterned weights, designed to mimic structures found in real-world data.
  • MULTI-MNIST: Datasets derived from MNIST, with patterned weights.
  • SYN-RANDOM: Datasets with randomly sampled weights, allowing for control over sparsity.

Additionally, a real-world dataset is used:

  • NeurIPS Titles Dataset: Used for topic modeling experiments, where paper titles are represented as binary vectors of word occurrences.

Key Insights and Ideas

  • Structure + Amortization = Robustness: The key insight is that combining model-specific structured posteriors (from classical methods) with amortized learning (via neural networks) leads to more robust inference, particularly when training data is limited. This structure helps prevent overfitting and reduces variance compared to pure AVI.
  • Tight Bounds vs. ELBO Maximization: The paper demonstrates that optimizing for the "tightest variational bound" (as in UB-CDI) does not necessarily lead to the best ELBO or posterior approximation. ACP's focus on maximizing the ELBO directly, even with a structured posterior, proves more effective.
  • Model-Dependent Inference is Helpful: Unlike standard AVI where the inference network is independent of the generative model, ACP's structured posterior inherently incorporates model knowledge. This "model-dependent posterior form" is shown to be beneficial for learning useful latent representations, especially in low-data scenarios.

Key Results

  • Improved Inference Accuracy: On synthetic datasets, ACP consistently outperforms or matches AVI, SVI, and CDI methods in terms of negative ELBO (NELBO), F1 scores, and Exact Match (EM) scores, especially when training data is limited. For instance, with only 20 training data points, ACP achieves a significantly better NELBO (22.2) compared to AVI (37.2) and SVI (22.6).
  • Better Parameter Estimation: ACP and AVI show comparable parameter recovery on SYN-PATTERN and MULTI-MNIST datasets when ample data is available. However, when data is scarce (e.g., 200 training points), ACP's parameter recovery degrades less severely than AVI's.
  • Effective Generative Modeling: In generative modeling tasks (SYN-RANDOM), ACP consistently outperforms AVI and SVI, maintaining good performance even with limited data and varying sparsity levels.
  • Document Classification: ACP's learned latent representations lead to higher classification accuracy on a news article dataset, especially with smaller training sets, and better visual separation of classes in t-SNE plots compared to AVI.

Limitations and Future Work

  • Computational Complexity: While ACP offers advantages, the paper notes that LB-CDI becomes computationally complex with large training sizes. The supplementary materials indicate that LB-CDI was not experimented with for large training sizes due to this.
  • Future Work: The authors suggest extending the ACP approach to hierarchical models, which would be a natural next step given its success in structured models.

Summary updated

Abstract

Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds. Modern approaches employ amortized variational inference, which uses a neural network to approximate any posterior without leveraging the structures of the generative models. In this paper, we propose Amortized Conjugate Posterior (ACP), a hybrid approach taking advantages of both types of approaches. Specifically, we use the classical methods to derive specific forms of posterior distributions and then learn the variational parameters using amortized inference. We study the effectiveness of the proposed approach on the noisy-or model and compare to both the classical and the modern approaches for approximate inference and parameter learning. Our results show that the proposed method outperforms or are at par with other approaches.
BibTeX
@inproceedings{yan2020amortized,
  author = {Yiming Yan and Melissa Ailem and Fei Sha},
  title = {{Amortized Inference of Variational Bounds for Learning Noisy-OR}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=Gn5CPFVZgBP},
}
Figure