Neuronized Priors for Bayesian Sparse Linear Regression

Minsuk Shin, Jun S. Liu

TL;DR

This paper introduces neuronized priors, a novel framework that reparameterizes Bayesian shrinkage priors using neural-network-inspired activation functions. By transforming Gaussian variables through functions like ReLU, the authors unify popular priors (Lasso, Horseshoe, Spike-and-Slab) into a single form that eliminates the need for slow latent indicator variables, resulting in significantly faster and more stable computation.

What Problem Does This Paper Address?

In high-dimensional regression, we want to identify a few important predictors out of thousands (variable selection). Bayesian methods are great for this because they quantify uncertainty, but they face a major "usability gap" compared to non-Bayesian methods like the Lasso:

  1. Computational Complexity: Classic "spike-and-slab" priors use discrete latent variables (0 or 1) to indicate if a variable is "in" the model. This creates a combinatorial explosion that makes Markov Chain Monte Carlo (MCMC) sampling notoriously slow.
  2. Lack of Unification: Different priors (Horseshoe, Laplace, Cauchy) require different specialized algorithms. There is no "one-stop-shop" framework to test multiple prior beliefs easily.
  3. Thresholding Issues: Continuous shrinkage priors (like the Horseshoe) never actually set coefficients to zero; they just make them very small, forcing researchers to use arbitrary thresholds to perform actual selection.

The authors aim to create a framework that is as flexible as the best Bayesian models but as computationally efficient as their frequentist counterparts.

Methods

The core innovation is the neuronized prior. Instead of defining a prior directly on the regression coefficient $\theta_j$, they reparameterize it as: $$\theta_j = T(\alpha_j - \alpha_0)w_j$$

  • $w_j$ (Weight): A standard Gaussian variable.
  • $\alpha_j$ (Scale): Another Gaussian variable.
  • $T(\cdot)$ (Activation Function): A non-decreasing function that determines the "shape" of the prior.
  • $\alpha_0$ (Bias): A hyperparameter that controls the overall sparsity level.

By simply changing the activation function $T$, you can emulate different classic priors:

  • ReLU ($max\{0, t\}$): Yields a Spike-and-Slab prior. Because ReLU is exactly zero for negative inputs, it provides explicit variable selection without needing binary latent indicators.
  • Identity ($t$): Approximates the Bayesian Lasso.
  • Exponential variants: Approximate heavy-tailed priors like the Horseshoe or Cauchy.

The authors also introduce CAAN (Coordinate-Ascent Algorithm for Neuronized priors), a scalable optimization routine for finding the "best" model (MAP estimate) much faster than full MCMC.

Datasets

The authors validated their approach using:

  • Synthetic Data: High-dimensional simulations ($p > n$) with varying degrees of correlation between predictors (Toeplitz designs).
  • Boston Housing Data: A classic benchmark for predicting median home values using 10 socio-economic variables.
  • Bardet-Biedl Syndrome Data: A real-world genomic dataset containing mRNA expression levels for 31,042 probes in rats. The goal was to find genes associated with the TRIM32 gene (linked to the syndrome) using a filtered set of $n=120$ samples and $p=200$ predictors.

Key Insights and Ideas

  • The "Latent-Free" Spike: The use of the ReLU function is a "lightbulb moment." It allows the model to sit exactly at zero (the spike) while remaining in a continuous parameter space. This avoids the "jumping" between discrete states that slows down traditional Bayesian selection.
  • Parameter Expansion: The product form ($weight \times scale$) acts as a form of "parameter expansion." In MCMC, this is known to improve the "mixing" of the chain, allowing the algorithm to explore the posterior distribution much more efficiently.
  • Theoretical "Free Lunch": The authors prove that this reparameterization doesn't sacrifice rigor. It achieves the optimal posterior contraction rate, meaning as you get more data, the model converges to the true underlying sparse signal at the fastest possible mathematical rate.

Key Results

  • Speed and Efficiency: In the gene expression study, the Neuronized Horseshoe (N-HS) was at least 2x more efficient (measured by Effective Sample Size per second) than the standard Horseshoe implementation.
  • Predictive Accuracy: On the Bardet-Biedl dataset, the Neuronized Spike-and-Slab (MAP version) achieved the lowest Mean Squared Prediction Error (MSPE) among all tested methods, including standard Lasso and SCAD.
  • Optimization Stability: The CAAN algorithm proved more robust than previous EM-based methods (like EMVS). While EMVS often got stuck in local optima depending on where it started, CAAN consistently found better, more stable models.

Limitations and Future Work

  • Bimodality: The current formulation has difficulty emulating "nonlocal" priors—priors that are explicitly bimodal and push coefficients away from zero.
  • Structured Sparsity: The paper primarily treats predictors as independent. The authors suggest that future work could extend this to multivariate neuronized priors to handle groups of variables (Group Lasso style) or spatial correlations.
  • Beyond Linear Models: While the paper focuses on linear regression, the "neuronization" logic is general. Extending this to logistic regression or survival analysis is a logical next step.

Software Availability: The authors have released an R package, NPrior, on CRAN for researchers to implement these methods.

Summary updated

Abstract

Although Bayesian variable selection methods have been intensively studied, their routine use in practice has not caught up with their non-Bayesian counterparts such as Lasso, likely due to difficulties in both computations and flexibilities of prior choices. To ease these challenges, we propose the neuronized priors to unify and extend some popular shrinkage priors, such as Laplace, Cauchy, horseshoe, and spike-and-slab priors. A neuronized prior can be written as the product of a Gaussian weight variable and a scale variable transformed from Gaussian via an activation function. Compared with classic spike-and-slab priors, the neuronized priors achieve the same explicit variable selection without employing any latent indicator variables, which results in both more efficient and flexible posterior sampling and more effective posterior modal estimation. Theoretically, we provide specific conditions on the neuronized formulation to achieve the optimal posterior contraction rate, and show that a broadly applicable MCMC algorithm achieves an exponentially fast convergence rate under the neuronized formulation. We also examine various simulated and real data examples and demonstrate that using the neuronization representation is computationally more or comparably efficient than its standard counterpart in all well-known cases. An R package NPrior is provided for using neuronized priors in Bayesian linear regression.
BibTeX
@article{shin2021neuronized,
  author = {Minsuk Shin and Jun S. Liu},
  title = {{Neuronized Priors for Bayesian Sparse Linear Regression}},
  journal = {Journal of the American Statistical Association},
  year = {2021},
  doi = {10.1080/01621459.2021.1876710},
}

Loading related content...

Figure