Variational Bridge Regression

Artin ArmaganAISTATS 2009

Abstract

This paper introduces Variational Bridge Regression (VBR), a computationally efficient Bayesian framework designed for sparse regression. By applying variational inference to the non-convex Bridge penalty (where the parameter $\gamma < 1$), VBR mimics the behavior of "best subset selection" while avoiding the computational pitfalls of traditional optimization. Unlike point-estimation methods like the Lasso, VBR provides a full approximate posterior distribution, enabling robust uncertainty quantification. Experimental results demonstrate that VBR significantly outperforms standard sparse regression techniques in model selection consistency and prediction accuracy, especially in high-dimensional settings.

Introduction

In high-dimensional data analysis, the primary goal is often to identify a small subset of variables that explain an outcome—a property known as sparsity. The "Bridge" regression framework generalizes this goal: while $\gamma = 2$ corresponds to Ridge regression and $\gamma = 1$ to the Lasso, setting $\gamma < 1$ produces even sparser and more accurate models. However, the resulting non-convex optimization problem is notoriously difficult to solve, often leading traditional algorithms to get stuck in local minima. While Markov Chain Monte Carlo (MCMC) methods can handle these priors, they are too computationally expensive for large-scale data. VBR bridges this gap by utilizing Variational Bayes to provide a fast, robust alternative.

Methodology

VBR employs a hierarchical Bayesian model where regression coefficients ($\beta$) are assigned an "exponential power" prior. To make the non-convex problem tractable, the author utilizes several key technical innovations:

  • Scale Mixture of Normals (SMN): The complex Bridge prior is represented as a mixture of normal distributions, allowing the model to treat the prior as a standard Gaussian with a latent variance component.
  • Mean Field Variational Approximation: Instead of slow simulations, VBR finds a simpler distribution that minimizes the Kullback-Leibler divergence to the true posterior.
  • The Moment Trick: Since the mixing distribution for latent variances is not explicitly known for all $\gamma$, the author derives the necessary expected moments ($\langle \tau_j \rangle$) by differentiating the integral representation of the prior.
  • Dynamic Pruning: To enhance speed and stability, the algorithm iteratively removes irrelevant predictors as their coefficients shrink toward zero.

Experimental Results

The performance of VBR was evaluated through rigorous synthetic simulation studies comparing it against OLS, Lasso, Adaptive Lasso, and the Non-negative Garrote.

Model Selection Consistency

A critical advantage of VBR is its ability to correctly identify the underlying sparse model. As shown in the comparative analysis of Model 1 and Model 2 (Table 1), VBR consistently achieves higher "Correct Model" (CM) percentages than Lasso-based methods. For instance, in Model 2 with $n=100$, VBR reaches 100% CM accuracy, whereas Lasso (CV) only identifies the correct model 20% of the time.

Table 1: Performance comparison of VBR against standard methods across different models and sample sizes. CM denotes the percentage of times the correct model was selected.

Robustness to Noise

VBR also demonstrates superior performance under varying noise levels. In scenarios with higher standard deviations ($\sigma=3$), VBR maintains significantly lower Mean Squared Error (MSE) and higher model selection accuracy compared to the Lasso and Adaptive Lasso, which often suffer from high false-positive rates (incorrectly identified variables, 'I').

Table 2: Performance metrics under different noise levels (σ=1 and σ=3). VBR maintains high CM rates even as noise increases.

Key Insights and Discussion

  • Uncertainty Quantification: By providing an approximate posterior rather than a single point estimate, VBR allows researchers to quantify confidence in specific variable inclusions.
  • The Bridge-RVM Connection: The author identifies a theoretical link between VBR (as $\gamma \to 0$) and Variational Relevance Vector Machines (VRVM), connecting Bridge regression to broader sparse Bayesian learning literature.
  • High-Dimensional Efficiency: In $p > n$ scenarios, VBR's pruning strategy ensures that computational burden decreases as the model converges on the true signals, preventing the inclusion of irrelevant variables that often plague the Lasso.

Limitations and Future Work

While VBR offers significant improvements, hyperparameter tuning for $\lambda\gamma$ currently requires empirical trial-and-error or cross-validation. Future research should focus on automated tuning methods and testing the framework on real-world biological or financial datasets where true sparsity levels are unknown. Additionally, extending the experimental validation to the Probit (binary) regression framework outlined in the paper remains a priority.

Summary updated

BibTeX
@article{armagan2009variational,
  author = {Artin Armagan},
  title = {{Variational Bridge Regression}},
  journal = {AISTATS},
  year = {2009},
}
Figure