TL;DR
This paper introduces BIVAS, a novel Bayesian method for bi-level variable selection that efficiently handles high-dimensional data by employing variational inference. BIVAS offers a scalable and computationally efficient alternative to traditional MCMC methods, demonstrating superior performance in variable selection, parameter estimation, and speed across simulations and real-world applications, including multi-task learning.
What Problem Does This Paper Address?
In many real-world scenarios, data variables naturally exhibit a hierarchical structure. For instance, genetic variants can be grouped by genes, or features in a multi-task learning problem can be shared across tasks. This structure implies that not only individual variables but also entire groups of variables might be relevant to the outcome. The challenge lies in simultaneously identifying which groups are important and which specific variables within those groups contribute to the model. Existing Bayesian methods, while powerful, often rely on computationally intensive Markov Chain Monte Carlo (MCMC) techniques that struggle with the scale of modern datasets. This paper aims to bridge this gap by developing a scalable and efficient Bayesian approach for this bi-level variable selection problem.
Methods
The core innovation of BIVAS lies in its use of variational inference to approximate the posterior distribution, moving away from computationally expensive MCMC. Instead of a standard mean-field approximation, BIVAS proposes a hierarchical factorization that leverages the specific structure of bi-level variable selection. This allows for a more accurate approximation of the posterior. To further enhance efficiency, the authors develop a computationally efficient and fully parallelizable variational expectation-maximization (EM) algorithm. The method is also extended to handle multi-task learning scenarios, where shared predictors across different tasks are treated as groups.
Datasets
The paper evaluates BIVAS on:
- Simulated datasets: Generated with varying levels of group and individual sparsity, signal-to-noise ratios (SNR), and correlation structures.
- Real-world datasets:
- GWAS data (HDL): From the Wellcome Trust Case Control Consortium (WTCCC) and Northern Finland Birth Cohort (NFBC), where SNPs are grouped by genes.
- IMDb movie review data: Used for multi-task learning, with movie genres (drama, comedy, horror, action, thriller, romance) serving as tasks and representative words as features.
- Rheumatoid Arthritis (RA) and Type 1 Diabetes (T1D) data: From the WTCCC, also analyzed for gene-level associations.
Key Insights and Ideas
- Variational Inference for Bi-Level Sparsity: Applying variational inference to bi-level variable selection is a key contribution, offering a scalable alternative to MCMC.
- Hierarchical Factorization: The proposed hierarchical factorization of the posterior distribution is tailored to the bi-level structure, leading to more accurate approximations than standard mean-field methods.
- Computational Efficiency and Parallelization: The development of a computationally efficient and parallelizable EM algorithm is crucial for handling large-scale datasets.
- Multi-Task Learning Extension: Adapting BIVAS for multi-task learning effectively leverages shared information across tasks, improving performance, especially for tasks with smaller sample sizes.
- Interpretable Parameters: The variational parameters offer intuitive interpretations, such as updated posterior inclusion probabilities for groups and variables.
Key Results
- Variable Selection Performance: BIVAS demonstrates competitive or superior performance in variable selection compared to existing methods like varbvs, cMCP, and GEL, particularly in scenarios with strong group sparsity and higher SNRs. It shows good control over False Discovery Rate (fdr).
- Estimation Accuracy: BIVAS achieves comparable or better estimation accuracy for coefficients compared to other methods, especially in challenging settings.
- Computational Efficiency: BIVAS is significantly faster than MCMC-based methods (e.g., BSGS-SS, using only ~1% of its computational time) and is comparable to or faster than other variational methods, benefiting greatly from parallelization.
- Multi-Task Learning: In multi-task learning, BIVAS outperforms varbvs and Lasso, especially when group sparsity is prominent, by effectively utilizing shared patterns across tasks.
- Real Data Applications: BIVAS successfully identifies relevant genes and SNPs in GWAS data and demonstrates strong predictive performance on the IMDb movie review dataset, highlighting its practical utility.
Limitations and Future Work
- Initial Value Sensitivity: The paper notes that the performance of the EM algorithm can be sensitive to the choice of initial values for the parameters, a common challenge in optimization-based methods.
- Approximation Accuracy: While variational inference provides efficiency, it is an approximation. The accuracy of parameter estimates might be affected, though posterior means of latent variables appear robust.
- Future Work: The authors suggest further exploration of the method's performance and extensions to more complex models. The availability of an R package ('bivas') and simulation code facilitates further research and application.
Summary updated