Parsimonious Bayesian deep networks

Mingyuan Zhou

TL;DR

Parsimonious Bayesian Deep Networks (PBDNs) use Bayesian nonparametrics to automatically infer the optimal width and depth of a neural network directly from data. By greedily adding layers and pruning unnecessary hidden units, PBDNs eliminate the need for manual architecture tuning and cross-validation while maintaining high accuracy and low computational costs for predictions.

The Challenge of Architecture Tuning

Designing deep neural networks (DNNs) usually involves a frustrating amount of manual tuning—adjusting the number of layers (depth) and hidden units (width) to find the sweet spot between underfitting and overfitting. While kernel-based methods like Support Vector Machines (SVMs) offer a more principled approach, they scale poorly; the number of support vectors often grows linearly with the training data size, making them slow and memory-intensive for large datasets.

This paper introduces a "parsimonious" middle ground: a model that has the representational power of a deep network but automatically determines its own complexity. The goal is a model that is "just right" for the data, requiring neither exhaustive architecture searches nor expensive fine-tuning.

Methods: Building the PBDN

The PBDN is built on two primary innovations:

  1. Infinite Support Hyperplane Machine (iSHM): This is the fundamental building block of the network. It is essentially a single-hidden-layer network that is theoretically "infinitely wide." To make it practical, the author uses a Gamma Process prior. This Bayesian nonparametric tool acts as a shrinkage mechanism, automatically "turning off" unnecessary hidden units (hyperplanes) so that only a finite, relevant number are actually used.
  2. Greedy Layer-wise Learning: Instead of training the whole deep network at once (which is difficult when the architecture is unknown), the PBDN grows one layer at a time. After a layer is trained, the model uses a forward model selection criterion—specifically the Akaike Information Criterion (AIC)—to decide if adding another layer will actually improve the model or just lead to overfitting. If the AIC stops improving, the network stops growing.

The model uses a Bernoulli-Poisson link (a generalization of the noisy-OR construction) to define class probabilities. Because this link is inherently asymmetric, the author trains iSHMs in pairs—one to capture the characteristics of class "1" and another for class "0"—to ensure the entire covariate space is well-covered.

Key Insights and Ideas

Convex Polytope Constraints

A single iSHM layer defines a decision boundary shaped like a convex polytope. While a single polytope cannot solve highly complex, non-linear problems on its own, stacking these layers allows the network to "warp" the data space. This effectively carves out highly complex, non-convex decision boundaries.

Evolution of decision boundaries on the Two Spirals dataset as pairs of iSHMs are added.

Automatic Sparsity

Unlike standard DNNs where you must choose a fixed layer size (e.g., 512 units), the Gamma Process in a PBDN naturally prunes the width. In many tests, the model found that only 5 or 6 units were necessary where a standard DNN might use hundreds.

Interpretable Subtypes

Because of the noisy-OR structure, the active hidden units in the first layer correspond to "data subtypes" or clusters near the decision boundary. This provides a level of interpretability often missing in deep learning, allowing users to visualize exactly what features the network is keying in on.

Visualizations of MNIST data subtypes identified by the PBDN and classification error rates compared to standard DNNs.

Key Results

  • Accuracy: PBDNs achieved state-of-the-art or competitive classification accuracy across benchmark datasets (including MNIST and various UCI datasets), often outperforming both tuned DNNs and RBF-kernel SVMs.
  • Efficiency: On the image dataset, an SVM required 212 support vectors, whereas the PBDN achieved better accuracy using an average of only 22 hyperplanes. This translates to significantly faster out-of-sample prediction speeds.
  • No Tuning Required: The greedy AIC-based strategy successfully determined the network depth automatically. For most MNIST tasks, the model correctly identified that a single layer was sufficient, whereas for more complex datasets like ijcnn1, it grew deeper.

Limitations and Future Work

  • Binary Focus: The current formulation is primarily designed for binary classification. While it can be extended, the "pair of iSHMs" approach would need to be scaled efficiently for multi-class problems.
  • Structured Data: For data with strong spatial or temporal correlations (like full images or audio), the linear hyperplanes should ideally be replaced with convolutional structures (CNN-iSHM). The author notes this as a promising direction for future research.
  • Greedy vs. Global: While greedy training is efficient and enables the AIC stopping criterion, it may miss global optima that end-to-end backpropagation might find in a fixed architecture.

Summary updated

Abstract

Combining Bayesian nonparametrics and a forward model selection strategy, we construct parsimonious Bayesian deep networks (PBDNs) that infer capacity-regularized network architectures from the data and require neither cross-validation nor fine-tuning when training the model. One of the two essential components of a PBDN is the development of a special infinite-wide single-hidden-layer neural network, whose number of active hidden units can be inferred from the data. The other one is the construction of a greedy layer-wise learning algorithm that uses a forward model selection criterion to determine when to stop adding another hidden layer. We develop both Gibbs sampling and stochastic gradient descent based maximum a posteriori inference for PBDNs, providing state-of-the-art classification accuracy and interpretable data subtypes near the decision boundaries, while maintaining low computational complexity for out-of-sample prediction.
BibTeX
@inproceedings{zhou2018parsimonious,
  author = {Mingyuan Zhou},
  title = {{Parsimonious Bayesian deep networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rkZjjD-_bH},
}
Figure