Structured Conditional Continuous Normalizing Flows for Efficient Amortized Inference in Graphical Models

Christian Weilbach, Boyan Beronov, Frank Wood, William HarveyAISTATS 2020 · OpenReview

Abstract

We exploit minimally faithful inversion of graphical model structures to specify sparse continuous normalizing flows (CNFs) for amortized inference. We find that the sparsity of this factorization, combined with augmented normalizing flows and a symmetrized Kullback-Leibler objective, enables state-of-the-art inference performance with significantly reduced computational costs compared to standard dense models.

Introduction

Continuous Normalizing Flows (CNFs) are powerful tools for density estimation because they model data transformations as a continuous path (an ODE) rather than a sequence of discrete layers. However, they face a significant "speed limit" due to high computational costs and difficulty scaling to high-dimensional data.

In the context of amortized inference—where a model is trained to quickly estimate hidden variables ($z$) given observations ($x$)—standard CNFs often rely on "black-box" neural networks that ignore the known dependency structures of the probabilistic model. This paper introduces Structured Conditional Continuous Normalizing Flows (SC-CNFs), which leverage the inherent structure of graphical models to create sparse, efficient, and accurate inference engines.

Methods

The authors propose three primary innovations to improve CNF efficiency and accuracy:

  1. Structural Sparsity: Instead of using fully connected layers, the CNF's underlying neural network uses a "minimally faithful inversion" of the graphical model. Weight matrices are masked so that neurons only communicate if there is a statistical dependency between the corresponding variables. This drastically reduces the number of parameters and floating-point operations (FLOPs).
  2. Augmented Normalizing Flows: The framework adds "nuisance variables" (extra dimensions) to each node in the graphical model. This provides the flow with more degrees of freedom, making it easier for the ODE solver to find a smooth path from the prior distribution to the complex posterior.
  3. Symmetrized KL Objective: While standard training minimizes either forward or reverse KL divergence, the authors utilize the Symmetrized KL (Jeffrey’s divergence). This combines both signals to provide a stronger learning gradient, leading to faster convergence and better handling of heavy-tailed distributions.
Overview of the SC-CNF construction process, from graphical model specification to the structured vector field of the flow.

Results and Discussion

The SC-CNF approach was evaluated on a complex synthetic arithmetic circuit and an MNIST stochastic deconvolution task.

Computational Efficiency

The SC-CNF required less than half the FLOPs of the state-of-the-art FFJORD model while achieving equal or better accuracy. Furthermore, the structured flows were numerically better conditioned; adaptive integrators required significantly fewer steps to reach a solution (approximately 50 steps vs. 75-100 for dense models).

Inference Performance

On the arithmetic circuit, the structured model learned significantly faster than fully connected baselines, which often plateaued at higher error rates. In the MNIST task, the method successfully learned to invert convolutional operators, producing diverse and accurate reconstructions of digits from blurred or noisy inputs.

Comparison of training convergence using expected symmetrized KL divergence across different sparsity patterns, highlighting the superior performance of the minimally faithful inverse.

Conclusion

By forcing neural networks to respect the structure of the underlying graphical model, SC-CNFs bake domain knowledge directly into the architecture. This acts as a powerful regularizer that improves both numerical stability and computational efficiency. Future work includes extending this framework to handle discrete variables and developing automated schemes for adaptive dimension augmentation.

Summary updated

Abstract

We exploit minimally faithful inversion of graphical model structures to specify sparse continuous normalizing flows (CNFs) for amortized inference. We find that the sparsity of this factorization ...
BibTeX
@inproceedings{weilbach2020structured,
  author = {Christian Weilbach and Boyan Beronov and Frank Wood and William Harvey},
  title = {{Structured Conditional Continuous Normalizing Flows for Efficient Amortized Inference in Graphical Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=YtVTt1ppYxm},
}
Figure