TL;DR
Parsimonious Bayesian Deep Networks (PBDNs) use Bayesian nonparametrics to automatically infer the optimal width and depth of a neural network directly from data. By greedily adding layers and pruning unnecessary hidden units, PBDNs eliminate the need for manual architecture tuning and cross-validation while maintaining high accuracy and low computational costs for predictions.
The Challenge of Architecture Tuning
Designing deep neural networks (DNNs) usually involves a frustrating amount of manual tuning—adjusting the number of layers (depth) and hidden units (width) to find the sweet spot between underfitting and overfitting. While kernel-based methods like Support Vector Machines (SVMs) offer a more principled approach, they scale poorly; the number of support vectors often grows linearly with the training data size, making them slow and memory-intensive for large datasets.
This paper introduces a "parsimonious" middle ground: a model that has the representational power of a deep network but automatically determines its own complexity. The goal is a model that is "just right" for the data, requiring neither exhaustive architecture searches nor expensive fine-tuning.
Methods: Building the PBDN
The PBDN is built on two primary innovations:
- Infinite Support Hyperplane Machine (iSHM): This is the fundamental building block of the network. It is essentially a single-hidden-layer network that is theoretically "infinitely wide." To make it practical, the author uses a Gamma Process prior. This Bayesian nonparametric tool acts as a shrinkage mechanism, automatically "turning off" unnecessary hidden units (hyperplanes) so that only a finite, relevant number are actually used.
- Greedy Layer-wise Learning: Instead of training the whole deep network at once (which is difficult when the architecture is unknown), the PBDN grows one layer at a time. After a layer is trained, the model uses a forward model selection criterion—specifically the Akaike Information Criterion (AIC)—to decide if adding another layer will actually improve the model or just lead to overfitting. If the AIC stops improving, the network stops growing.
The model uses a Bernoulli-Poisson link (a generalization of the noisy-OR construction) to define class probabilities. Because this link is inherently asymmetric, the author trains iSHMs in pairs—one to capture the characteristics of class "1" and another for class "0"—to ensure the entire covariate space is well-covered.
Key Insights and Ideas
Convex Polytope Constraints
A single iSHM layer defines a decision boundary shaped like a convex polytope. While a single polytope cannot solve highly complex, non-linear problems on its own, stacking these layers allows the network to "warp" the data space. This effectively carves out highly complex, non-convex decision boundaries.
Automatic Sparsity
Unlike standard DNNs where you must choose a fixed layer size (e.g., 512 units), the Gamma Process in a PBDN naturally prunes the width. In many tests, the model found that only 5 or 6 units were necessary where a standard DNN might use hundreds.
Interpretable Subtypes
Because of the noisy-OR structure, the active hidden units in the first layer correspond to "data subtypes" or clusters near the decision boundary. This provides a level of interpretability often missing in deep learning, allowing users to visualize exactly what features the network is keying in on.
Key Results
- Accuracy: PBDNs achieved state-of-the-art or competitive classification accuracy across benchmark datasets (including MNIST and various UCI datasets), often outperforming both tuned DNNs and RBF-kernel SVMs.
- Efficiency: On the image dataset, an SVM required 212 support vectors, whereas the PBDN achieved better accuracy using an average of only 22 hyperplanes. This translates to significantly faster out-of-sample prediction speeds.
- No Tuning Required: The greedy AIC-based strategy successfully determined the network depth automatically. For most MNIST tasks, the model correctly identified that a single layer was sufficient, whereas for more complex datasets like ijcnn1, it grew deeper.
Limitations and Future Work
- Binary Focus: The current formulation is primarily designed for binary classification. While it can be extended, the "pair of iSHMs" approach would need to be scaled efficiently for multi-class problems.
- Structured Data: For data with strong spatial or temporal correlations (like full images or audio), the linear hyperplanes should ideally be replaced with convolutional structures (CNN-iSHM). The author notes this as a promising direction for future research.
- Greedy vs. Global: While greedy training is efficient and enables the AIC stopping criterion, it may miss global optima that end-to-end backpropagation might find in a fixed architecture.
Summary updated