bayesreg: Bayesian Regression Models with Global-Local Shrinkage Priors

Daniel F. Schmidt, Enes MakalicOpenalex Unknown 2016 · DOI · OpenAlex

Abstract

The bayesreg R package fits linear or generalized linear regression models using Bayesian global-local shrinkage prior hierarchies, as described in Polson and Scott (2010). It provides an efficient implementation of ridge, lasso, horseshoe, and horseshoe+ regression for various target distributions, including logistic, Gaussian, Laplace, Student-t, Poisson, and geometric. The package utilizes algorithms summarized in Makalic and Schmidt (2016) to ensure high performance and numerical stability.

Introduction

In modern data science, "high-dimensional" problems—where the number of predictors ($P$) is large relative to the number of observations ($N$)—are increasingly common. Standard regression techniques often struggle with overfitting in these scenarios. While frequentist methods like the Lasso are popular for sparse recovery, they lack natural uncertainty estimates. Bayesian shrinkage priors offer a principled alternative for feature selection and uncertainty quantification, but they have historically been difficult to implement and computationally demanding.

The bayesreg package addresses these challenges by providing a user-friendly "toolbox" for state-of-the-art Bayesian analysis. It supports a wide array of data types and offers built-in tools for variable selection, outlier robustness, and multi-core parallelization.

Methods and Implementation

The package implements several sophisticated Bayesian techniques centered around global-local shrinkage priors. These include Ridge, Lasso, Horseshoe, and Horseshoe+ priors. The Horseshoe family is particularly effective at aggressively shrinking noise coefficients while preserving significant signals.

To ensure efficiency, bayesreg employs Gibbs sampling. For high-dimensional matrices, it dynamically switches between Rue’s algorithm and the algorithm by Bhattacharya et al., depending on the dimensions of $N$ and $P$. For Generalized Linear Models (GLMs), it uses specialized samplers: the Polya-gamma sampler for logistic regression and a gradient-assisted Metropolis-Hastings algorithm for count data (Poisson and Geometric).

Core input parameters for the bayesreg function, including model distribution and prior selection.

Robustness to outliers is achieved through Laplace and Student-t error models. Additionally, version 1.3 introduced automatic multi-core parallelization, significantly reducing run-times for large-scale MCMC simulations.

Key Insights and Diagnostics

Beyond basic model fitting, bayesreg provides several advanced features for model comparison and interpretation:

  • Automatic Factor Grouping: Categorical variables are automatically grouped, and shrinkage is applied to the set as a whole.
  • Model Selection via WAIC: The package calculates the Widely Applicable Information Criterion (WAIC), allowing for quantitative comparison between different distributional assumptions.
  • Variable Importance Ranking: Covariates are ranked by their importance, aiding in feature engineering and model understanding.
Summary of the statistical outputs returned by the bayesreg function, including WAIC and variable importance ranks.

The package also provides diagnostic metrics such as Effective Sample Size (ESS) and posterior t-statistics, ensuring users can assess MCMC convergence and variable significance with clarity.

Conclusion

The bayesreg package offers a robust framework for researchers and engineers to embrace Bayesian modeling in high-dimensional spaces. While users must remain mindful of MCMC tuning and potential memory constraints for extremely large datasets, the package's versatility and efficiency make it a powerful tool for moving beyond simple point estimates to full posterior inference.

Summary updated

Abstract

Fits linear or generalized linear regression models using Bayesian global-local shrinkage prior hierarchies as described in Polson and Scott (2010) &lt;<a href="https://doi.org/10.1093%2Facprof%3Aoso%2F9780199694587.003.0017" target="_top">doi:10.1093/acprof:oso/9780199694587.003.0017</a>&gt;. Provides an efficient implementation of ridge, lasso, horseshoe and horseshoe+ regression with logistic, Gaussian, Laplace, Student-t, Poisson or geometric distributed targets using the algorithms summarized in Makalic and Schmidt (2016) &lt;<a href="https://doi.org/10.48550%2FarXiv.1611.06649" target="_top">doi:10.48550/arXiv.1611.06649</a>&gt;.
BibTeX
@inproceedings{schmidt2016bayesreg,
  author = {Daniel F. Schmidt and Enes Makalic},
  title = {{bayesreg: Bayesian Regression Models with Global-Local Shrinkage Priors}},
  booktitle = {Openalex Unknown},
  year = {2016},
  doi = {10.32614/cran.package.bayesreg},
}
Figure