Introduction
Deep Generative Models (DGMs), particularly Variational Autoencoders (VAEs), are powerful tools for discovering latent factors in high-dimensional data. However, they suffer from a fundamental theoretical limitation: unidentifiability. Because neural networks are highly flexible, they can learn many different latent representations that explain the observed data equally well. This makes it difficult to determine if the learned "factors" represent true underlying drivers or are merely mathematical artifacts. While previous approaches achieved identifiability by assuming independent latent factors or using auxiliary labels, these conditions are often violated in real-world scenarios. This paper proposes the Sparse VAE, which achieves identifiability by imposing structural constraints on the decoder.
Methods: Sparse Decoding and Structural Constraints
The core innovation is the Sparse DGM. Unlike standard models where every latent factor can influence every observed feature, the Sparse DGM enforces that each feature depends only on a small subset of factors.
To implement this, the authors introduce:
- Sparse Decoding: A per-feature masking mechanism that restricts the influence of latent factors.
- Spike-and-Slab Lasso (SSL) Prior: A mixture prior consisting of a "spike" at zero to eliminate irrelevant factors and a "slab" to allow significant factors to have large weights.
- Anchor Feature Assumption: The theoretical proof of identifiability relies on the existence of "anchor features"—data dimensions that depend on only one specific latent factor. The model identifies these automatically during training.
Datasets and Evaluation
The Sparse VAE was evaluated on several datasets:
- Synthetic Data: To test ground-truth recovery with known correlations.
- PeerRead: Word counts from scientific abstracts.
- MovieLens: User-movie ratings for top films.
- Zeisel: RNA molecule counts in mouse cortex cells.
- Semi-synthetic PeerRead: Testing generalization under varying factor correlations.
Key Results
The Sparse VAE demonstrates a superior ability to recover true latent factors compared to standard VAEs, particularly when factors are correlated.
- Factor Recovery: In synthetic experiments, the Sparse VAE achieved significantly higher DCI (Disentanglement, Completeness, and Informativeness) scores. As shown in the figure above, the Sparse VAE's estimated weights (W) closely match the true weights, whereas the standard VAE fails to recover the sparse structure.
- Generalization and Likelihood: On real-world datasets like MovieLens, the Sparse VAE achieved lower held-out Negative Log-Likelihood (170.9 vs. 175.9 for standard VAE).
- Interpretability: The model identified cleaner, more distinct topics in MovieLens (e.g., separating Sci-Fi from 80s adventure) and achieved 95% accuracy in predicting cell types in the Zeisel genomics data.
Limitations and Future Work
Despite its strengths, the Sparse VAE has limitations. It is currently optimized for tabular data where features have fixed meanings, making it less suitable for image data. Additionally, the per-feature masking introduces computational overhead, requiring $G$ forward passes through the decoder. Finally, while the model is theoretically identifiable, the non-convex nature of neural network training means that finding the global optimum remains an optimization challenge.
Summary updated