Abstract
We introduce Group Spike-and-slab Variational Bayes (GSVB), a scalable method for group sparse regression. A fast coordinate ascent variational inference (CAVI) algorithm is developed for several common model families, including Gaussian, Binomial, and Poisson. Theoretical guarantees for our proposed approach are provided by deriving contraction rates for the variational posterior in grouped linear regression. Through extensive numerical studies, we demonstrate that GSVB provides state-of-the-art performance, offering a computationally inexpensive substitute to MCMC while performing comparably or better than existing MAP methods. Additionally, we analyze three real-world datasets wherein we highlight the practical utility of our method, demonstrating that GSVB provides parsimonious models with excellent predictive performance, variable selection, and uncertainty quantification.
Introduction
In modern high-dimensional statistics, features often exhibit natural clustering. In genomics, for instance, single nucleotide polymorphisms (SNPs) are frequently grouped by biological pathways or gene locations. Standard sparse regression techniques, such as the LASSO, treat variables in isolation, potentially ignoring these underlying structures. Group sparse regression addresses this by selecting or discarding entire blocks of variables simultaneously.
However, researchers typically face a stark trade-off between two paradigms:
- Markov Chain Monte Carlo (MCMC): The gold standard for accuracy and uncertainty quantification (providing credible intervals), but computationally prohibitive for high-dimensional datasets.
- Maximum A Posteriori (MAP) / Frequentist Methods: Computationally efficient but limited to point estimates, offering little to no information regarding the confidence of the selection.
Group Spike-and-Slab Variational Bayes (GSVB) bridges this gap, providing a scalable framework that maintains the rigor of Bayesian uncertainty quantification at a fraction of the computational cost of MCMC.
The GSVB Framework
Model and Prior
GSVB utilizes a "group spike-and-slab" prior. This prior is a mixture of two components:
- The Spike: A Dirac mass at zero that forces entire groups of coefficients to be exactly zero, facilitating sparse selection.
- The Slab: A multivariate double exponential distribution that allows for group-wise shrinkage and captures the magnitude of active groups.
Variational Inference and CAVI
To perform inference, the authors developed a fast Coordinate Ascent Variational Inference (CAVI) algorithm. This algorithm is tailored for Gaussian, Binomial, and Poisson likelihoods, making it applicable to a wide range of regression and classification tasks.
A key innovation in GSVB is the comparison of two variational families:
- Family $Q$: A standard mean-field family assuming independence between all variables in the posterior.
- Family $Q'$: A more flexible family that allows for unrestricted covariance within groups. This modification allows the model to better capture dependencies between related features, leading to more realistic uncertainty estimates.
Theoretical Guarantees
Unlike many heuristic variational approaches, GSVB is supported by formal theoretical analysis. The authors derive posterior contraction rates, proving that as the sample size increases, the GSVB variational posterior converges to the true underlying parameter at an optimal rate. This provides a rigorous foundation for using variational approximations in grouped settings.
Empirical Performance
Simulation Studies
The method was benchmarked against MCMC and state-of-the-art MAP methods. In Gaussian simulations with 1,000 variables, the speed disparity was stark:
- GSVB: 1.4 to 10 seconds.
- MCMC: 37 to 51 minutes.
Despite this speed, GSVB-B (the version utilizing the $Q'$ family) performed almost identically to MCMC in terms of variable selection accuracy and predictive performance. Furthermore, GSVB provided marginal credible sets with impressive coverage, meaning the calculated "error bars" reliably contained the true parameter values.
Real-World Applications
The practical utility of GSVB was demonstrated across three biological datasets:
- Mouse Genome Database: Predicting LDL cholesterol levels using over 10,000 SNPs ($p \gg n$).
- Bardet-Biedl Syndrome Study: Identifying gene expressions associated with genetic disease ($n=120$).
- MEMset Splice Site Detection: A classification task identifying DNA motifs.
In these applications, GSVB consistently produced more parsimonious models. For example, in the splice site detection task, GSVB achieved comparable AUC and F-scores to the Group LASSO while selecting only 10-12 groups, compared to the 60 groups selected by the Group LASSO. This sparsity makes the resulting models significantly easier for practitioners to interpret.
Discussion and Future Work
The Importance of Intra-group Correlation
A primary insight of this work is that the standard mean-field assumption is often too restrictive for grouped data. By allowing for intra-group correlation via the $Q'$ family, GSVB provides significantly better uncertainty quantification and larger, more realistic credible sets than simpler variational approximations.
Limitations
While GSVB represents a significant step forward, the authors note several areas for future development:
- Initialization: Like many variational methods, the algorithm can be sensitive to starting values. Using a Group LASSO solution as an initial point is recommended to avoid local optima.
- Variance Underestimation: Although $Q'$ is an improvement, Variational Bayes still tends to slightly underestimate posterior variance.
- Bi-level Selection: The current model selects or rejects entire groups. Future extensions could incorporate "sparse-group" priors to allow for the selection of specific important variables within a chosen group.
Conclusion
GSVB offers a powerful new tool for high-dimensional regression. By combining the speed of variational inference with the structural advantages of group spike-and-slab priors, it provides a computationally efficient alternative to MCMC that does not sacrifice the ability to quantify uncertainty or select interpretable, parsimonious models.
Summary updated