D-MFVI: Distributed Mean Field Variational Inference using Bregman ADMM

Behnam Babagholami-Mohamadabadi, Sejong Yoon, Vladimir PavlovicarXiv 2015 · OpenReview

TL;DR

This paper introduces D-MFVI, a novel framework for distributed Bayesian inference that allows complex models to be learned across networks of devices. By combining Mean Field Variational Inference with Bregman ADMM, it efficiently handles distributed data and computation, enabling accurate parameter estimation and uncertainty quantification in scenarios where centralized processing is infeasible.

What Problem Does This Paper Address?

Modern applications, especially those involving sensor networks, generate massive amounts of data that are often distributed across numerous devices. Centralizing this data for traditional machine learning algorithms is frequently impractical due to communication constraints, sheer data volume, and privacy concerns. Furthermore, many existing distributed approaches focus on point estimates (deterministic models) and fail to capture the uncertainty inherent in Bayesian modeling, which is crucial for tasks like online learning and robust decision-making. This paper addresses the challenge of performing distributed Bayesian inference, enabling the learning of full posterior distributions for model parameters across a network of devices.

Methods

The core contribution is the D-MFVI (Distributed Mean Field Variational Inference) framework. It leverages:

  • Mean Field Variational Inference (MFVI): This is a standard technique for approximating intractable posterior distributions by assuming a simpler, factorized form for the approximate posterior. In MFVI, all latent variables are assumed to be independent.
  • Bregman Alternating Direction Method of Multipliers (B-ADMM): This is a generalized version of ADMM, which is a powerful optimization technique for distributed problems with consensus constraints. B-ADMM replaces the standard quadratic penalty terms in ADMM with Bregman divergences, allowing it to better exploit the structure of the problem and often leading to closed-form updates.

The D-MFVI framework applies MFVI locally at each node in a network. To enforce consensus on global parameters across nodes, it uses B-ADMM. The key idea is to impose equality constraints on the posterior parameters of the global variables across different nodes. The paper demonstrates that by using conjugate priors and likelihoods, the updates for the variational parameters can often be computed in closed form.

Datasets

The paper evaluates its proposed D-MFVI framework on the following:

  • Synthetic data: Used for empirical convergence analysis of the D-BPCA algorithm.
  • Caltech 3D Objects on Turntable dataset: Used for evaluating D-BPCA in Structure from Motion (SfM) tasks.
  • Hopkins155 dataset: Also used for evaluating D-BPCA in SfM tasks.
  • Netflix dataset: Used for evaluating D-BPCA in a large-scale Matrix Factorization (MF) problem.

The paper notes that for the Caltech and Hopkins datasets, they used subsets and setups similar to previous work [2], [15].

Key Insights and Ideas

  • Bridging the gap between distributed computation and Bayesian inference: The paper successfully demonstrates a general framework for performing distributed Bayesian learning, which is a significant step towards enabling complex probabilistic models in decentralized systems.
  • Bregman ADMM for efficient distributed variational inference: Using B-ADMM proves to be an effective way to solve the constrained optimization problem arising from distributed MFVI, often leading to closed-form updates and faster convergence.
  • Uncertainty quantification in distributed settings: Unlike many deterministic distributed methods, D-MFVI inherently provides uncertainty estimates for model parameters, which is crucial for many real-world applications.
  • Flexibility of the framework: The proposed D-MFVI framework is general and can be applied to various probabilistic models, as demonstrated by its application to Bayesian PCA (BPCA) for Structure from Motion (SfM) and Matrix Factorization (MF).

Key Results

  • Structure from Motion (SfM): D-BPCA consistently outperformed D-PPCA (a previous distributed probabilistic PCA method) on both synthetic and real-world datasets (Caltech, Hopkins155) in terms of subspace angle error, especially in the presence of missing data (MAR and MNAR). D-BPCA also showed robustness to network topology and parameter choices.
  • Matrix Factorization (MF): For the large-scale Netflix dataset, D-BPCA achieved competitive results (lower RMSE) compared to centralized approaches, while offering the benefits of distributed computation, reduced storage, and parallel processing.
  • Online Learning: The paper highlights that D-BPCA naturally supports online Bayesian estimation in distributed sensing networks, with its accuracy closely following centralized BPCA.
  • Convergence: Empirical results showed that D-BPCA converges efficiently (within 10² iterations) and achieves a global solution equivalent to centralized methods, regardless of network topology or the number of nodes.

Limitations and Future Work

  • Conjugate Priors: The current framework relies on conjugate priors and exponential family distributions for closed-form updates. The authors acknowledge that this limits its applicability to non-conjugate models.
  • Future Work: The authors suggest extending the framework to more advanced variational inference algorithms such as collapsed variational inference and structured variational inference to handle non-conjugate models and potentially improve performance. They also mention that generalizing D-BPCA to handle missing data is straightforward.

Summary updated

Abstract

Bayesian models provide a framework for probabilistic modelling of complex datasets. However, many of such models are computationally demanding especially in the presence of large datasets. On the other hand, in sensor network applications, statistical (Bayesian) parameter estimation usually needs distributed algorithms, in which both data and computation are distributed across the nodes of the network. In this paper we propose a general framework for distributed Bayesian learning using Bregman Alternating Direction Method of Multipliers (B-ADMM). We demonstrate the utility of our framework, with Mean Field Variational Bayes (MFVB) as the primitive for distributed Matrix Factorization (MF) and distributed affine structure from motion (SfM).
BibTeX
@misc{babagholamimohamadabadi2015d,
  author = {Behnam Babagholami-Mohamadabadi and Sejong Yoon and Vladimir Pavlovic},
  title = {{D-MFVI: Distributed Mean Field Variational Inference using Bregman ADMM}},
  year = {2015},
  eprint = {1507.00824},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1507.00824},
}
Figure