Deep Regression Bayesian Network and Its Applications

Siqi Nie, Meng Zheng, Qiang Ji

Abstract

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this paper, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian Networks due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods often through an auxiliary network to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of max-sum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark datasets of different models are given for both data representation and feature learning tasks.

Introduction: The "Explaining Away" Challenge

Most contemporary deep learning models are either purely discriminative or rely on simplified generative frameworks. While models like Generative Adversarial Networks (GANs) produce high-quality imagery, they often lack a rigorous probabilistic framework for handling uncertainty. Traditional probabilistic models, such as Deep Belief Networks (DBNs) or Deep Boltzmann Machines (DBMs), frequently assume that latent variables are independent given the data to maintain computational efficiency.

However, this assumption ignores the "explaining away" principle—a fundamental concept in Bayesian logic where multiple causes for an observation become dependent once that observation is known. Modeling these dependencies typically makes calculating the posterior probability of hidden states (inference) "intractable," requiring an exponential number of calculations. This research introduces the Deep Regression Bayesian Network (DRBN) to bridge this gap, creating a deep, fully directed model that preserves complex latent dependencies.

Comparison of deep generative model structures: (a) fully directed (DRBN/SBN), (b) hybrid (DBN), and (c) fully undirected (DBM).

Methodology

The DRBN architecture is constructed by stacking layers of Regression Bayesian Networks (RBNs). Unlike standard Restricted Boltzmann Machines (RBMs), the energy function derived for RBNs includes an explicit term for relationships among latent variables.

Max-Max Learning

To address the intractability of summing over all possible latent configurations (the "max-sum" approach), the authors propose a "max-max" operation. This strategy leverages the observation that posterior distributions in high-dimensional latent spaces are often extremely sparse, with probability mass concentrated on a few configurations. By identifying the single most likely configuration ($h^*$), the model can be optimized efficiently without exhaustive summation.

Augmented Coordinate Ascent (AugCA)

For Maximum A Posteriori (MAP) inference, the paper introduces Augmented Coordinate Ascent (AugCA). This hybrid method uses a feed-forward inference network to generate a rapid initial guess, which is then refined through coordinate ascent—iteratively updating latent variables one at a time until convergence. This approach balances the speed of feed-forward networks with the precision of iterative refinement.

The hierarchical structure of a Deep Regression Bayesian Network (DRBN) showing directed dependencies between latent layers.

Experimental Evaluation

The DRBN was evaluated across several benchmark datasets, including MNIST for log-likelihood accuracy, the Berkeley Data Set for image restoration, and various face databases (Multi-PIE, CAS-PEAL) for reconstruction tasks.

Key Results

  • Representation Accuracy: On the MNIST dataset, the DRBN achieved a log-likelihood of -170.75, nearly matching the exact mathematical value of -170.54. In contrast, standard variational approaches lagged significantly at -206.06, demonstrating that preserving dependencies is critical for representational power.
  • Image Restoration: In inpainting tasks on the Berkeley dataset, the DRBN using AugCA achieved a Peak Signal-to-Noise Ratio (PSNR) of 33.46, substantially outperforming traditional methods like GMM (25.71) and KSVD (24.13).
  • Feature Learning: For head pose estimation (BU Head Pose and MPII Gaze), the model achieved errors comparable to state-of-the-art systems despite using extremely low-resolution inputs (15x15 pixels).

Discussion and Future Work

The success of the max-max approach confirms that finding the "peak" of a distribution is often sufficient for high-quality learning in sparse, high-dimensional spaces. However, the iterative nature of coordinate ascent remains more computationally demanding than simple forward passes in CNNs. Future research will focus on scaling these optimizations to continuous latent spaces and conducting broader comparisons against modern generative architectures like Variational Autoencoders (VAEs) and Diffusion models.

Summary updated

Abstract

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this paper, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian Networks due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods often through an auxiliary network to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of max-sum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark datasets of different models are given for both data representation and feature learning tasks.
BibTeX
@inproceedings{nie2018deep,
  author = {Siqi Nie and Meng Zheng and Qiang Ji},
  title = {{Deep Regression Bayesian Network and Its Applications}},
  booktitle = {IEEE Signal Process. Mag},
  year = {2018},
  url = {https://openreview.net/forum?id=gmitr2eKtV1},
}

Loading related content...

Figure