Abstract
Deep belief networks are a powerful way to model complex probability distributions. However, it is difficult to learn the structure of a belief network, particularly one with hidden units. The Indian buffet process has been used as a nonparametric Bayesian prior on the structure of a directed belief network with a single infinitely wide hidden layer. Here, we introduce the cascading Indian buffet process (CIBP), which provides a prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network framework to allow each unit to vary its behavior between discrete and continuous representations. We use Markov chain Monte Carlo for inference in this model and explore the structures learned on image data.
Introduction
In the early 2010s, the rise of deep learning brought a significant challenge: "architecture engineering." Designers had to manually specify the number of layers and hidden units, often relying on trial and error or computationally expensive cross-validation. While learning weights was standard, inferring the underlying graph structure—especially with latent variables—remained difficult. This research addresses this by proposing a nonparametric Bayesian approach where the network's depth, width, and connectivity are learned directly from the data.
The Cascading Indian Buffet Process (CIBP)
The core innovation is the Cascading Indian Buffet Process (CIBP). While the standard Indian Buffet Process (IBP) allows for a single hidden layer of infinite width, the CIBP extends this recursively. In this framework, the features (or "dishes") activated in one layer serve as the "customers" for the subsequent layer.
Theoretically, this allows for infinite depth and width. However, the authors utilize the Foster-Lyapunov criterion to prove that the number of active units in each layer eventually decreases, ensuring the network naturally terminates and remains computationally tractable. This "rich-get-richer" dynamic encourages feature sharing across the network while maintaining sparsity.
Flexible Unit Representations
To enhance model flexibility, the CIBP is paired with Nonlinear Gaussian Belief Networks (NLGBN). Unlike traditional networks that fix unit types (e.g., all sigmoid or all ReLU), this model treats the identity of each unit as a latent variable. By adjusting a precision parameter ($\nu$), a unit can smoothly transition between behaving like a discrete binary switch and a continuous Gaussian variable. This allows the data to determine whether a feature is best represented by its presence/absence or its intensity.
Experimental Results
The model was evaluated on image datasets including Olivetti Faces, MNIST, and Frey Faces using Markov chain Monte Carlo (MCMC) for inference.
Feature Discovery
On the Olivetti Faces dataset, the model successfully learned sparse, interpretable features. The bottom layer of the discovered structures captured localized facial components such as mouths, noses, and eyebrows, rather than global, holistic patterns.
Structural Adaptation
The model demonstrated an ability to adapt its architecture to the nature of the data:
- MNIST: Learned highly sparse, local features.
- Frey Faces: Learned denser, more global connections to account for the high pixel correlation in video frames of a single subject.
- Depth: Across datasets, the MCMC sampler typically converged on structures with approximately three hidden layers, despite the prior allowing for much greater depth.
Discussion and Limitations
The CIBP offers a principled way to automate architecture design, moving away from manual tuning. However, the reliance on MCMC inference presents a significant computational bottleneck, requiring hours of CPU time for relatively small datasets. While the model excels at density modeling and unsupervised feature learning, scaling it to modern, massive datasets and competitive supervised performance remains a challenge for future work.
Summary updated