Abstract
Deep Generative Models (DGMs) often require manual tuning of latent space dimensionality, leading to sub-optimal performance if the capacity is poorly matched to data complexity. This paper presents IBP-Deep Generative Models (IBP-DGM), a nonparametric approach that integrates the Indian Buffet Process (IBP) to automatically infer the optimal number of latent features. By employing Black-Box Variational Inference (BBVI) to handle discrete latent variables, the proposed framework achieves significant improvements in semi-supervised classification tasks across complex image datasets, effectively pruning redundant dimensions and enhancing robustness to noise.
Introduction
Variational Autoencoders and other Deep Generative Models have revolutionized high-dimensional data representation. However, selecting the appropriate number of latent variables remains a "Goldilocks" challenge: too few variables fail to capture data complexity, while too many lead to overfitting or wasted capacity. Traditionally, this is addressed via exhaustive cross-validation.
While nonparametric Bayesian methods like the Indian Buffet Process (IBP) have long been used in linear models to discover latent structure, their integration into non-linear deep learning architectures has been hindered by optimization difficulties. This work introduces the IBP-DGM, which marries nonparametric flexibility with the representational power of deep networks.
Methodology
The IBP-DGM framework introduces a structured latent space that allows for data-driven dimensionality discovery.
Latent Factorization and the IBP Prior
The model decomposes the latent representation $z$ into a product of two components: $z = \tilde{z} \cdot \hat{z}$. Here, $\tilde{z}$ represents a continuous vector capturing feature content, while $\hat{z}$ is a binary indicator vector acting as an "on/off" switch for each feature. An Indian Buffet Process prior is placed on $\hat{z}$, allowing the model to theoretically access an infinite number of features while ensuring that only a finite, relevant subset is active for any given data point.
Inference and Learning
To optimize the model, the authors utilize Black-Box Variational Inference (BBVI). Standard reparameterization tricks are inapplicable to the discrete binary switches of the IBP. BBVI overcomes this by using the "log-derivative trick" and control variates, providing stable gradient estimates for backpropagation. For semi-supervised learning, the likelihood function is extended to incorporate both labeled and unlabeled data, encouraging the latent space to cluster in alignment with class labels.
Experimental Evaluation
The IBP-DGM was evaluated on several computer vision benchmarks under a semi-supervised setup (1% labeled data):
- MNIST & Rotated MNIST: Standard and rotated handwritten digits.
- MNIST + Background (Images/Random): Digits superimposed on complex backgrounds.
- NORB: 3D objects under varying conditions.
The model was tested using two distinct architectures: standard Dense Layers (DL) and Memory Networks (MN).
Results and Discussion
The IBP-DGM consistently outperformed the parametric "M2" baseline across all benchmarks.
As shown in the results for dense layers, the IBP-DGM significantly reduced error rates, particularly on challenging datasets like Rotated MNIST (from 38.80% to 32.82%) and MNIST with background images (from 16.16% to 8.99%).
When using Memory Network architectures, the performance gains persisted. The attention mechanisms in MNs appeared complementary to the IBP prior; while the IBP pruned unnecessary features, the MN focused on the most relevant aspects of the remaining active features.
Automatic Dimensionality Discovery
A key insight from the experiments is the model's ability to "prune" its own latent space. Even when initialized with a capacity of 50 dimensions, the IBP-DGM typically converged to using only 10-12 active components, effectively filtering out noise and identifying the intrinsic dimensionality of the data.
Conclusion
The IBP-DGM offers a robust solution to the latent dimensionality problem in deep generative modeling. By leveraging nonparametric Bayesian priors, the model achieves superior performance in semi-supervised tasks and demonstrates remarkable resilience to noise. While the discrete optimization requires more training epochs, the resulting models are more efficient and interpretable in their use of latent capacity. Future work will explore applications to heteroscedastic and temporal data.
Summary updated