The Deep Regression Bayesian Network and Its Applications: Probabilistic Deep Learning for Computer Vision

Siqi Nie, Meng Zheng, Qiang Ji

TL;DR

This paper introduces Deep Regression Bayesian Networks (DRBNs), a fully probabilistic directed generative model that preserves complex dependencies between latent variables. By replacing standard variational approximations with a "max-max" learning strategy and an augmented coordinate ascent inference method, the authors achieve superior data representation and competitive performance in tasks like image restoration and pose estimation.

What Problem Does This Paper Address?

Most deep learning models are either purely discriminative (like CNNs) or use simplified generative frameworks. While Generative Adversarial Networks (GANs) produce realistic images, they often lack a grounded probabilistic framework to handle uncertainty.

Existing probabilistic models like Deep Belief Networks (DBNs) or Deep Boltzmann Machines (DBMs) often make a major sacrifice for the sake of computational efficiency: they assume latent variables are independent given the data. This ignores the "explaining away" principle—a core tenet of Bayesian logic where multiple causes for an observation become dependent once that observation is known.

The challenge is that modeling these dependencies makes inference (calculating the posterior probability of hidden states) "intractable," requiring an exponential number of calculations. This paper seeks to bridge that gap, creating a model that is both deep and fully directed without falling into the trap of over-simplifying latent dependencies.

Methods

The authors propose the Deep Regression Bayesian Network (DRBN), which stacks layers of Regression Bayesian Networks (RBNs).

Key technical innovations include:

  • The Energy Function: The authors derive an energy function for RBNs that includes an explicit term for relationships among latent variables, distinguishing it from the standard Restricted Boltzmann Machine (RBM).
  • Max-Max Learning: Instead of the traditional "max-sum" approach (which requires summing over all possible latent configurations), they use a "max-max" operation. This is based on the insight that the posterior distribution in these models is often very sparse; the probability mass is concentrated on a few configurations. By finding the single most likely configuration ($h^*$), they can optimize the model efficiently.
  • Augmented Coordinate Ascent (AugCA): To find the most likely hidden state (MAP inference), the authors combine two worlds. They use a feed-forward "inference network" to provide a fast initial guess, then refine that guess using "coordinate ascent"—iteratively updating one latent variable at a time until convergence.

Datasets

The model was evaluated across a diverse range of benchmarks:

  • MNIST: Binarized handwritten digits for evaluating log-likelihood accuracy.
  • Berkeley Data Set: Used for image inpainting and restoration tasks.
  • Multi-PIE & CAS-PEAL: Large-scale face databases for denoising, reconstruction, and image generation.
  • UnityEye: Synthetic eye images for reconstruction.
  • BU Head Pose & MPII Gaze: Used to test the model's "feature learning" capabilities for regression tasks (predicting angles and coordinates).

Key Insights and Ideas

  • Preserving Dependencies Matters: The "max-max" method produces log-likelihoods significantly closer to the "exact" mathematical truth than standard variational methods. This suggests that the common practice of assuming independent latent variables (factorization) significantly weakens a model's representational power.
  • Sparsity is a Feature: The success of the max-max approach confirms that in high-dimensional latent spaces, we don't need to account for every possible state. Finding the "peak" of the distribution is often sufficient for high-quality learning.
  • Hybrid Inference is the "Sweet Spot": Purely iterative methods (like coordinate ascent) are accurate but slow and prone to local optima. Purely feed-forward methods (like inference networks) are fast but imprecise. Combining them (AugCA) provides a robust path to accurate inference.

Key Results

  • Representation Accuracy: On the MNIST dataset, the DRBN's log-likelihood (-170.75) was nearly identical to the exact calculated value (-170.54), whereas the standard variational approach lagged far behind (-206.06).
  • Image Restoration: In image inpainting tasks on the Berkeley dataset, the DRBN (AugCA) achieved a Peak Signal-to-Noise Ratio (PSNR) of 33.46, vastly outperforming traditional methods like GMM (25.71) and KSVD (24.13).
  • Robustness to Resolution: For head pose estimation, the DRBN achieved errors comparable to state-of-the-art models despite using extremely low-resolution input (15x15 pixels) and no 3D information.

Limitations and Future Work

While DRBNs show great promise, the authors note some hurdles:

  • Computational Cost: Even with the "max-max" optimization, coordinate ascent is more computationally demanding than the single forward pass of a standard CNN.
  • Scaling: The paper primarily focuses on binary latent variables. Extending these specific "max-max" optimizations to continuous latent spaces or very high-dimensional categorical variables remains a challenge for future research.
  • Comparison to Modern Generative SOTA: While the paper demonstrates superiority over DBNs and DBMs, further comparison against modern Variational Autoencoders (VAEs) and Diffusion models would be necessary to see where DRBNs sit in the current rapidly evolving landscape.

Summary updated

Abstract

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this article, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian networks (BNs) due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods, often through an auxiliary network, to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of maxsum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent (AugCA) are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark data sets of different models are given for both data representation and feature-learning tasks.
BibTeX
@inproceedings{nie2018deep,
  author = {Siqi Nie and Meng Zheng and Qiang Ji},
  title = {{The Deep Regression Bayesian Network and Its Applications: Probabilistic Deep Learning for Computer Vision}},
  booktitle = {IEEE Signal Process. Mag},
  year = {2018},
  url = {https://openreview.net/forum?id=gmitr2eKtV1},
}

Loading related content...

Figure