Abstract
With the flourishing development of high-dimensional data, sparse versions of principal component analysis (PCA) have imposed themselves as simple, yet powerful ways of selecting relevant features. GSPPCA (Globally Sparse Probabilistic PCA) is a new Bayesian framework that enforces a common sparsity pattern across all principal components. By deriving a closed-form expression for the marginal likelihood using multivariate Bessel distributions, the authors provide a principled way to perform unsupervised feature selection that is more interpretable and effective than standard sparse PCA methods.
Introduction
Principal Component Analysis (PCA) serves as a cornerstone for dimension reduction and data visualization. However, its standard formulation produces principal components that are linear combinations of all original variables, complicating the identification of specific features—such as genes or pixels—that drive the data's structure.
While 'Sparse PCA' variants address this by forcing many coefficients to zero, they typically focus on local sparsity, where each principal component identifies its own unique set of active variables. This lack of consensus across components can hinder the identification of a single set of relevant features for the entire dataset. To resolve this, the authors propose global sparsity: a model where all principal components share a common sparsity pattern, effectively performing feature selection for the entire subspace simultaneously.
Methodology
GSPPCA is built upon the Probabilistic PCA (PPCA) framework, which treats PCA as a generative latent variable model ($x = Wy + \epsilon$). To enforce global sparsity, the authors introduce a binary vector $v$ that acts as a gatekeeper for the rows of the loading matrix $W$. If $v_i = 0$, the $i$-th variable is excluded from all principal components.
Exact Marginal Likelihood and Model Selection
A primary technical contribution of this work is the derivation of an exact analytical expression for the marginal likelihood. By considering a 'noiseless limit' for active variables, the authors link the model to the multivariate Bessel distribution. This closed-form solution enables principled model selection via 'Occam’s Razor'—selecting the number of features that maximizes the evidence while naturally penalizing excessive complexity.
To navigate the $2^p$ possible combinations of the binary vector $v$, the authors employ a continuous relaxation ($u \in [0, 1]^p$) and a Variational Expectation-Maximization (VEM) algorithm. This approach identifies a candidate path of models, which are then evaluated using the exact marginal likelihood formula to determine the optimal sparsity level.
Experimental Results
The GSPPCA framework was evaluated on synthetic simulations and real-world datasets, including the Leukemia gene expression dataset and variations of the MNIST handwritten digit dataset.
Robustness and Variance
On the Leukemia dataset (Golub et al., 1999), which contains expression levels of 3,051 genes for 38 patients, GSPPCA (using $d=30$ components) retained significantly more variance for a given number of features compared to 'local' sparse PCA methods. This suggests that global sparsity is not just more interpretable, but also more efficient at capturing the dataset's overall structure. Simulations further confirmed that the method remains highly accurate even when significant noise is present, demonstrating robustness beyond the 'noiseless' assumption used in the mathematical derivation.
Interpretability in Complex Backgrounds
The advantages of global sparsity are particularly evident in the MNIST experiments. The model was tested on digits superimposed with random noise (mnist-back-rand) and natural image patches (mnist-back-image).
As illustrated above, standard sparse PCA (SPCA) often fails to distinguish between the digit and complex background noise. In contrast, GSPPCA's global constraint forces the model to focus on pixels that consistently contribute to the digit's shape across the entire latent subspace, resulting in significantly cleaner and more interpretable feature selection.
Discussion and Conclusion
GSPPCA provides a robust, Bayesian framework for global feature selection in PCA. By maintaining orthogonality and providing a closed-form solution for model selection, it offers significant advantages in interpretability over local sparsity methods.
While the current derivation relies on a noiseless limit for active variables, empirical results suggest the model is highly robust. Future work may explore fully 'noisy' closed-form solutions and automated estimation of both sparsity levels and latent dimensions. Despite the computational efficiency of the VEM algorithm, scaling to extremely high-dimensional datasets remains an area for further optimization.
Summary updated