Abstract
Although Bayesian variable selection methods have been intensively studied, their routine use in practice has not caught up with their non-Bayesian counterparts such as Lasso, likely due to difficulties in both computations and flexibilities of prior choices. To ease these challenges, we propose the neuronized priors to unify and extend some popular shrinkage priors, such as Laplace, Cauchy, horseshoe, and spike-and-slab priors. A neuronized prior can be written as the product of a Gaussian weight variable and a scale variable transformed from Gaussian via an activation function. Compared with classic spike-and-slab priors, the neuronized priors achieve the same explicit variable selection without employing any latent indicator variables, which results in both more efficient and flexible posterior sampling and more effective posterior modal estimation. Theoretically, we provide specific conditions on the neuronized formulation to achieve the optimal posterior contraction rate, and show that a broadly applicable MCMC algorithm achieves an exponentially fast convergence rate under the neuronized formulation. We also examine various simulated and real data examples and demonstrate that using the neuronization representation is computationally more or comparably efficient than its standard counterpart in all well-known cases. An R package NPrior is provided for using neuronized priors in Bayesian linear regression.
Introduction
In high-dimensional regression analysis, identifying a sparse set of significant predictors from a large pool of candidates is a fundamental challenge. While frequentist methods like the Lasso have gained widespread adoption due to their computational efficiency, Bayesian variable selection methods offer a more rigorous framework for uncertainty quantification. However, Bayesian approaches face a significant "usability gap" characterized by high computational complexity—often requiring slow Markov Chain Monte Carlo (MCMC) sampling over discrete latent spaces—and a lack of unified frameworks that allow researchers to easily switch between different prior beliefs.
This paper introduces neuronized priors, a novel framework that reparameterizes Bayesian shrinkage priors using neural-network-inspired activation functions. By transforming Gaussian variables through specific functions, this approach unifies popular priors (such as Lasso, Horseshoe, and Spike-and-Slab) into a single, continuous form that eliminates the need for discrete latent indicator variables.
Methodology: The Neuronized Framework
The core innovation lies in the reparameterization of the regression coefficient $\theta_j$. Instead of defining a prior directly on $\theta_j$, the authors propose:
$$\theta_j = T(\alpha_j - \alpha_0)w_j$$
Where:
- $w_j$ (Weight): A standard Gaussian variable.
- $\alpha_j$ (Scale): A Gaussian variable representing the scale.
- $T(\cdot)$ (Activation Function): A non-decreasing function determining the prior's shape.
- $\alpha_0$ (Bias): A hyperparameter controlling the overall sparsity level.
By selecting different activation functions $T$, the framework can emulate various classic priors. For instance, using the ReLU function ($max\{0, t\}$) yields a Spike-and-Slab prior. Because ReLU is exactly zero for negative inputs, it provides explicit variable selection (the "spike") while remaining in a continuous parameter space. This avoids the combinatorial explosion associated with traditional binary latent indicators. Other functions, such as the identity or exponential variants, approximate the Bayesian Lasso or heavy-tailed priors like the Horseshoe and Cauchy.
To facilitate efficient estimation, the authors also introduce CAAN (Coordinate-Ascent Algorithm for Neuronized priors), a scalable optimization routine designed to find the Maximum A Posteriori (MAP) estimate more robustly than traditional Expectation-Maximization (EM) methods.
Theoretical and Computational Properties
The neuronized formulation provides several advantages:
- Latent-Free Selection: The use of ReLU allows the model to achieve exact sparsity without jumping between discrete states, which typically slows down MCMC chains.
- Improved Mixing: The product form ($weight \times scale$) acts as a form of parameter expansion. In MCMC, this expansion is known to improve the "mixing" of the chain, allowing for more efficient exploration of the posterior distribution.
- Optimal Contraction: The authors prove that this reparameterization achieves the optimal posterior contraction rate, ensuring that the model converges to the true underlying sparse signal at the fastest possible mathematical rate as data volume increases.
Empirical Results
The framework was validated across synthetic and real-world datasets, including the Boston Housing benchmark and the Bardet-Biedl Syndrome genomic dataset ($n=120, p=200$).
- Computational Efficiency: In genomic studies, the Neuronized Horseshoe (N-HS) was found to be at least twice as efficient as standard implementations when measured by Effective Sample Size (ESS) per second.
- Predictive Performance: On the Bardet-Biedl dataset, the Neuronized Spike-and-Slab (via MAP estimation) achieved the lowest Mean Squared Prediction Error (MSPE) compared to Lasso and SCAD.
- Stability: The CAAN algorithm demonstrated superior stability over previous EM-based methods (like EMVS), which are often sensitive to initialization and prone to local optima.
Discussion and Future Work
While neuronized priors offer a powerful unification of Bayesian sparsity, some limitations remain. The current formulation is primarily designed for unimodal priors and may struggle to emulate "nonlocal" priors that are explicitly bimodal. Furthermore, the current implementation treats predictors as independent.
Future research directions include extending the framework to handle structured sparsity (such as Group Lasso) and applying the neuronization logic to non-linear models, including logistic regression and survival analysis. The authors have made the NPrior R package available on CRAN to support further adoption and research.
Summary updated