Probabilistic Graphical Models with Neural Networks in InferPy

Rafael Cabañas, Javier Cózar, Antonio Salmerón, Andrés R. MasegosaPGM 2020 · OpenReview

Abstract

InferPy is an open-source Python package for variational inference in probabilistic models containing neural networks. Other similar libraries are often difficult for non-expert users. InferPy provides a much more compact and simple way to code such models, at the expense of slightly reducing expressibility and flexibility. The main objective of this package is to permit its use without having a strong theoretical background or thorough knowledge of the deep learning frameworks.

Introduction

In modern machine learning, a significant divide exists between Probabilistic Graphical Models (PGMs) and Deep Learning. PGMs excel at reasoning under uncertainty and incorporating domain expertise through structured dependencies. Conversely, deep neural networks (NNs) are unparalleled at capturing complex, non-linear patterns in high-dimensional data. Bridging these two paradigms—Deep Probabilistic Modeling—offers the benefits of both: uncertainty quantification and high-capacity representation.

While frameworks like Pyro, Stan, and Edward2 facilitate this integration, they often present a steep learning curve. These tools require deep expertise in variational inference and complex programming abstractions. InferPy addresses this barrier by providing a high-level Python library designed for accessibility. Built as a user-friendly wrapper around TensorFlow Probability (TFP), InferPy allows researchers to define hybrid models using a simple, Keras-like syntax, prioritizing developer productivity over granular flexibility.

Methodology

InferPy functions as an abstraction layer over TensorFlow Probability and Edward2. Its design philosophy centers on a "P-Model" and "Q-Model" approach, where users define a generative model (prior and likelihood) and a variational model (approximating distribution) respectively.

Probabilistic Graphical Model (PGM) with integrated Neural Networks.

The library simplifies several complex tasks:

  • Plateau Notation Management: Using decorators like @inf.probmodel and context managers like with inf.datamodel(), InferPy automatically handles the replication of variables across data points.
  • Keras Integration: Neural networks are defined using standard tf.keras.Sequential objects, allowing seamless integration of existing deep learning architectures into probabilistic frameworks.
  • Automated Inference: Stochastic Variational Inference (SVI) is automated, enabling the simultaneous learning of global parameters and local hidden variables via backpropagation.

By defining variables on a "single instance" basis within a data model context, InferPy manages batching logic internally, reducing the common confusion surrounding tensor shapes in probabilistic programming.

Case Study: Variational Autoencoders

The utility of InferPy is demonstrated through a classic Variational Autoencoder (VAE) task using the MNIST dataset. The model architecture involves:

  1. A latent space ($z$) representing digit features.
  2. A decoder NN ($NN_1$) that generates image data ($x$) from the latent space.
  3. A secondary NN ($NN_2$) using a categorical distribution for classification ($y$).
  4. An encoder NN (the Q-model) to perform inference on the latent variables.

This hybrid approach allows the neural networks to define the parameters of probability distributions (e.g., the mean and variance of a Normal distribution), effectively combining PGM structure with NN expressivity.

Results and Discussion

InferPy significantly reduces the "boilerplate" code required for deep probabilistic models. A complex hierarchical model involving multiple neural networks and probability distributions can be implemented in approximately 30-40 lines of code. The API simplifies posterior sampling and predictive checks through intuitive methods like p.posterior("z", ...).sample().

However, this simplicity comes with trade-offs. InferPy is inherently less flexible than the low-level libraries it abstracts. Researchers requiring highly customized inference algorithms may find the high-level abstractions limiting. Future development aims to expand support for inference methods beyond SVI, such as Markov Chain Monte Carlo (MCMC), and improve diagnostic tools for model convergence.

Conclusion

InferPy democratizes deep probabilistic modeling by providing a compact, intuitive interface for combining PGMs and Neural Networks. By hiding the underlying mathematical and programming complexities, it enables a broader range of researchers to leverage uncertainty quantification in deep learning applications.

Summary updated

Abstract

InferPy is an open-source Python package for variational inference in probabilistic models containing neural networks. Other similar libraries are often difficult for non-expert users. InferPy provides a much more compact and simple way to code such models, at the expense of slightly reducing expressibility and flexibility. The main objective of this package is to permit its use without having a strong theoretical background or thorough knowledge of the deep learning frameworks.
BibTeX
@inproceedings{cabanas2020probabilistic,
  author = {Rafael Cabañas and Javier Cózar and Antonio Salmerón and Andrés R. Masegosa},
  title = {{Probabilistic Graphical Models with Neural Networks in InferPy}},
  booktitle = {PGM},
  year = {2020},
  url = {https://openreview.net/forum?id=8upwF9iuXD},
}
Figure