Learning the Structure of Deep Sparse Graphical Models

Ryan P. Adams, Hanna Wallach, Zoubin Ghahramani

Abstract

Deep belief networks are a powerful way to model complex probability distributions. However, learning the structure of a belief network, particularly one with hidden units, is difficult. The Indian buffet process has been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single infinitely wide hidden layer. In this paper, we introduce the cascading Indian buffet process (CIBP), which provides a nonparametric prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network so each unit can additionally vary its behavior between discrete and continuous representations. We provide Markov chain Monte Carlo algorithms for inference in these belief networks and explore the structures learned on several image data sets.

Introduction

A central challenge in deep learning is architecture selection. Determining the optimal number of layers and hidden units typically requires extensive manual tuning or computationally expensive searches. While the Indian Buffet Process (IBP) has successfully allowed for "infinite" width in single-layer models—letting data dictate the number of features—extending this nonparametric flexibility to deep, multi-layered architectures has remained difficult. The primary hurdle is defining a prior over architectures that is theoretically infinite in both dimensions yet guaranteed to produce finite, tractable structures for inference.

The Cascading Indian Buffet Process (CIBP)

The authors propose the Cascading Indian Buffet Process (CIBP) as a solution. Building on the standard IBP metaphor—where customers (data points) select dishes (features) from an infinite buffet—the CIBP stacks these processes. In this "cascade," the features selected at layer $m$ become the "customers" for the next buffet at layer $m+1$.

Network structures generated by CIBP with varying hyperparameters.

This recursive structure allows the network to grow in both depth and width. To ensure the model remains finite, the authors employ the Foster-Lyapunov stability criterion to prove that the process eventually terminates. As shown in the drift analysis, the expected number of new features decreases as the depth increases, eventually hitting zero.

Convergence of network width over depth, showing the expected drift toward zero.

Nonlinear Gaussian Belief Networks

The CIBP is paired with Nonlinear Gaussian Belief Networks (NLGBN). A key innovation here is the use of a precision parameter that allows individual hidden units to adaptively switch between discrete binary-like behavior and continuous Gaussian-like behavior. This enables the model to capture diverse data characteristics within a single unified framework.

Experimental Results and Insights

The model was evaluated on several image datasets, including Olivetti Faces, MNIST Digits, and Frey Faces. The CIBP successfully inferred data-specific architectures:

  • MNIST: Learned sparse, local features such as strokes and blobs, typically resulting in a three-layer structure (approx. 120, 100, and 70 units).
  • Frey Faces: Discovered denser, more global connections to capture subtle variations in facial expressions.
  • Olivetti Faces: Inferred interpretable facial components like eyes, noses, and glasses.

In missing-data tasks, the model demonstrated strong generative capabilities, producing realistic reconstructions of the bottom halves of images given only the top halves.

Conclusion and Future Work

The CIBP offers a principled Bayesian approach to learning network architecture. While the current implementation relies on computationally intensive Markov Chain Monte Carlo (MCMC) methods—taking several hours for inference—it provides a valuable tool for discovering the intrinsic complexity of data. Future work may involve using these nonparametric priors to inform the design of more efficient discriminative models or exploring faster inference techniques.

Summary updated

Abstract

Deep belief networks are a powerful way to model complex probability distributions. However, learning the structure of a belief network, particularly one with hidden units, is difficult. The Indian buffet process has been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single infinitely wide hidden layer. In this paper, we introduce the cascading Indian buffet process (CIBP), which provides a nonparametric prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network so each unit can additionally vary its behavior between discrete and continuous representations. We provide Markov chain Monte Carlo algorithms for inference in these belief networks and explore the structures learned on several image data sets.
BibTeX
@inproceedings{adams2010learning,
  author = {Ryan P. Adams and Hanna Wallach and Zoubin Ghahramani},
  title = {{Learning the Structure of Deep Sparse Graphical Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=KEnOqS7uLY},
}
Figure