Introduction
Deep Latent Generative Models (DLGMs), such as Variational Autoencoders (VAEs), have become a cornerstone of unsupervised representation learning. However, these models typically yield dense latent representations where every feature is assigned a non-zero value. In contrast, many real-world phenomena are inherently sparse—only a small subset of potential features or concepts is relevant to any given observation. While traditional methods employ soft regularization (e.g., $L_1$ penalties) to encourage sparsity, they often lack the flexibility to adapt the degree of sparsity to individual data points or to enforce hard constraints required for resource-constrained environments like IoT devices.
This paper introduces the Sparse Deep Latent Generative Model (SDLGM), a framework that treats the sparsity level of a representation as a learnable, per-observation latent variable.
The SDLGM Framework
The core innovation of SDLGM is a hierarchical sampling strategy that explicitly separates the modeling of "which" features are active from "what" those features represent.
Two-Step Discrete Sampling
The model employs a two-step process to generate a sparse representation for an observation:
- Sparsity Modeling: For each observation, the model draws $L_0$ binary "gate" variables $\omega_{i, \ell}$ from a Bernoulli distribution. The sum of these gates defines the sparsity level (the number of active components) for that specific instance.
- Feature Modeling: Simultaneously, $L_0$ independent categorical features $z_{i, \ell}$ are sampled. The final representation is the element-wise product of these features and their corresponding gates, effectively masking out inactive components.
To maintain end-to-end differentiability despite the discrete nature of these variables, the authors utilize the Gumbel-Softmax distribution. This continuous approximation allows the model to be trained using standard backpropagation.
Experimental Results
The SDLGM was evaluated on both unsupervised tasks (MNIST) and multi-label supervised tasks (Birds, Emotions, CAL500).
Unsupervised Representation Learning
On the MNIST dataset, SDLGM achieved a Negative Evidence Lower Bound (ELBO) of 99.05, outperforming standard Gumbel-Softmax (101.5) and other discrete latent variable models like DARN and MuProp. The model successfully learns to cluster digits in the latent space while adapting the number of active features to the complexity of the input.
Multi-label Classification and Sparsity Prediction
In supervised settings, SDLGM variants (Discriminative, Generative, and Conditional) consistently outperformed baseline models like Sparsemax and Logistic regression in terms of Micro- and Macro-averaged F1 scores. Furthermore, in a custom multi-digit MNIST task designed to test counting capabilities, SDLGM predicted the frequency of specific features with a Mean Squared Error (MSE) of 0.05, significantly better than the Random Forest baseline (0.16).
Conclusion
SDLGM provides a robust framework for learning sparse representations where the sparsity level is an emergent property of the data rather than a fixed hyperparameter. By leveraging differentiable discrete sampling, the model achieves state-of-the-art performance in both generative and discriminative tasks. Future work includes scaling this approach to high-dimensional data and exploring its application in sparse attention mechanisms for natural language processing.
Summary updated