varbvs: Fast Variable Selection for Large-scale Regression

Peter Carbonetto, Xiang Zhou, Matthew StephensarXiv 2017 · DOI · arXiv

Abstract

We introduce varbvs, a suite of functions written in R and MATLAB for regression analysis of large-scale data sets using Bayesian variable selection methods. We have developed numerical optimization algorithms based on variational approximation methods that make it feasible to apply Bayesian variable selection to very large data sets. With a focus on examples from genome-wide association studies, we demonstrate that varbvs scales well to data sets with hundreds of thousands of variables and thousands of samples, and has features that facilitate rapid data analyses. Moreover, varbvs allows for extensive model customization, which can be used to incorporate external information into the analysis. We expect that the combination of an easy-to-use interface and robust, scalable algorithms for posterior computation will encourage broader use of Bayesian variable selection in areas of applied statistics and computational biology.

Introduction: The High-Dimensional Challenge

In fields like genomics, researchers frequently encounter "high-dimensional" problems where the number of potential predictors (e.g., genetic markers) far exceeds the number of samples. Identifying the specific variables that drive an outcome is known as the variable selection problem.

Traditionally, researchers have relied on penalized regression methods like the Lasso or Elastic Net. While these methods are computationally efficient, they typically provide only "point estimates"—binary indicators of whether a variable is included in the model—without a clear measure of uncertainty. In contrast, Bayesian Variable Selection (BVS) is considered the gold standard because it provides a "Posterior Inclusion Probability" (PIP) for every variable, representing the percentage chance that a variable belongs in the model. However, BVS has historically been limited by the computational intensity of Markov Chain Monte Carlo (MCMC) simulations, which can take days or weeks on massive datasets.

Methods: Variational Approximation

The varbvs package addresses this bottleneck by replacing slow MCMC sampling with variational approximation. This technique transforms a complex integration problem into a faster numerical optimization task.

Key technical features include:

  • Spike-and-Slab Priors: A robust Bayesian framework where a variable's effect is either exactly zero (the "spike") or drawn from a normal distribution (the "slab").
  • Fully-Factorized Approximation: To ensure tractability for big data, the importance of each variable is assumed to be independent given the hyperparameters.
  • Automatic Hyperparameter Estimation: The software uses EM-like algorithms to learn optimal settings, such as the expected number of relevant variables, directly from the data.

Comparative Analysis and Datasets

The authors evaluated varbvs across three biological datasets of increasing scale. For smaller datasets like the Leukemia gene expression data (72 patients, 3,571 genes), varbvs was compared directly against the popular glmnet package, which implements traditional penalized regression.

Figure 4: Standard Lasso diagnostics (via glmnet) for the Leukemia dataset, showing cross-validation error, sparsity, and coefficient paths. varbvs aims to provide richer uncertainty measures than these point-estimate paths.

While traditional methods like those shown above provide a single "best" path for coefficients, varbvs allows for model averaging, which accounts for uncertainty in the selection process. This is particularly evident in the medium-scale Outbred Mice GWAS (993 mice, 79,748 SNPs), where varbvs identified a single SNP with a PIP of 1.000, offering a much clearer signal than the noisy p-values generated by traditional association tests.

Key Results and Scalability

The primary breakthrough of varbvs is its ability to handle massive datasets without sacrificing the benefits of the Bayesian approach.

  • Performance on Large-Scale Data: In an analysis of a Crohn’s disease dataset containing 442,001 variables, varbvs completed the task in approximately 39 hours on a single CPU—a feat previously considered impossible for standard BVS methods.
  • Accuracy: Cross-validation on mouse data showed that varbvs prediction errors were 97% correlated with results from much slower MCMC-based methods, proving that the variational approximation is a highly reliable surrogate for full posterior sampling.
  • Biological Discovery: The tool successfully pinpointed known risk loci for Crohn's disease (e.g., NOD2 and IL23R) and identified specific genetic markers for testis weight in mice.

Limitations and Future Directions

Despite its speed, varbvs faces challenges common to optimization-based methods. The underlying problem is non-convex, meaning results can occasionally depend on the initial "guess" or variable ordering. The authors recommend using multiple random seeds to ensure stability. Additionally, the "fully-factorized" assumption can sometimes cause the model to pick only one representative from a group of highly correlated variables.

Future development for varbvs focuses on implementing even faster optimization schemes, such as quasi-Newton methods and SQUAREM acceleration, to further push the boundaries of million-variable regression analysis.

Summary updated

Abstract

We introduce varbvs, a suite of functions written in R and MATLAB for regression analysis of large-scale data sets using Bayesian variable selection methods. We have developed numerical optimization algorithms based on variational approximation methods that make it feasible to apply Bayesian variable selection to very large data sets. With a focus on examples from genome-wide association studies, we demonstrate that varbvs scales well to data sets with hundreds of thousands of variables and thousands of samples, and has features that facilitate rapid data analyses. Moreover, varbvs allows for extensive model customization, which can be used to incorporate external information into the analysis. We expect that the combination of an easy-to-use interface and robust, scalable algorithms for posterior computation will encourage broader use of Bayesian variable selection in areas of applied statistics and computational biology. The most recent R and MATLAB source code is available for download at Github (https://github.com/pcarbo/varbvs), and the R package can be installed from CRAN (https://cran.r-project.org/package=varbvs).
BibTeX
@misc{carbonetto2017varbvs,
  author = {Peter Carbonetto and Xiang Zhou and Matthew Stephens},
  title = {{varbvs: Fast Variable Selection for Large-scale Regression}},
  year = {2017},
  eprint = {1709.06597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.06597},
}
Figure