VBLinLogit: Variational Bayesian linear and logistic regression

Jan DrugowitschJ. Open Source Softw 2019 · DOI · DBLP

Abstract

Standard linear and logistic regression models often struggle with high-dimensional or sparse datasets, leading to overfitting and poor generalization. While Bayesian methods offer a robust alternative through uncertainty estimation and regularization, traditional Markov Chain Monte Carlo (MCMC) approaches are often computationally prohibitive. We present VBLinLogit, a lightweight MATLAB/Octave library that utilizes Variational Bayesian (VB) inference to provide a fast, scalable solution for high-dimensional regression problems. VBLinLogit incorporates Automatic Relevance Determination (ARD) for feature selection and eliminates the need for cross-validation by optimizing model evidence.

Introduction

In the era of high-dimensional data, researchers frequently encounter "underdetermined" problems where the number of predictors ($p$) significantly exceeds the number of observations ($n$). In such settings, classical frequentist regressions often fail to generalize. Bayesian approaches mitigate these issues by incorporating prior information and providing a full posterior distribution of the parameters. However, the computational cost of MCMC sampling—the standard for Bayesian inference—often limits its applicability in large-scale or real-time scenarios.

VBLinLogit addresses this gap by providing an implementation of Variational Bayesian inference. By treating inference as an optimization problem rather than a sampling problem, VB offers a computationally efficient "middle ground" that maintains the benefits of a probabilistic framework while achieving speeds comparable to point-estimation methods.

Methodology

VBLinLogit is designed for simplicity and efficiency, requiring no external dependencies beyond core MATLAB or Octave.

Variational Bayesian Inference

The core of the library is the VB algorithm, which approximates the true posterior distribution by minimizing the Kullback-Leibler (KL) divergence between the approximation and the true posterior. This optimization-based approach is significantly faster than MCMC, particularly as dimensionality increases.

Automatic Relevance Determination (ARD)

A key feature of the library is its implementation of ARD. ARD uses hierarchical priors to automatically determine the relevance of each input feature. By placing a prior on the precision of the regression coefficients, the model can effectively "prune" irrelevant variables by driving their weights to zero. This provides a Bayesian alternative to the Lasso (L1 regularization) but with the added benefit of providing uncertainty estimates for the remaining features.

Model Evidence and Regularization

Unlike frequentist methods that require a separate validation set to tune regularization parameters (e.g., $\lambda$ in Ridge or Lasso), the Bayesian framework handles regularization through the marginal likelihood, or model evidence. VBLinLogit optimizes this evidence directly, allowing for robust model selection and hyperparameter tuning using only the training data.

Applications

The utility of VBLinLogit has been validated through its application in several peer-reviewed studies:

  • Neuroscience: It has been used to quantify information within neural populations (Kanitscheider et al., 2015).
  • Cognitive Science: Researchers have employed the library to model human decision-making processes and reinforcement learning (Oh et al., 2016; Wang & Rehder, 2017).
  • Finance: It has been applied to the prediction of corporate financial distress (Ruxanda et al., 2018).

Additionally, the library's impact is reflected in its integration into the Probabilistic Modeling Toolkit (pmtk3), a widely used resource accompanying Kevin Murphy's textbook, Machine Learning: A Probabilistic Perspective.

Discussion and Limitations

VBLinLogit provides a "full Bayesian" treatment of ARD, meaning it estimates the posterior distribution of hyperparameters rather than relying on point estimates (Type-II Likelihood). This approach offers a more complete picture of model uncertainty.

However, users should be aware that no single regularization method is superior in all contexts. The performance of ARD versus alternatives like the Bayesian Lasso or Stochastic Search Variable Selection (SSVS) depends on the sparsity structure of the underlying data. Furthermore, while MATLAB has introduced native Bayesian regression functions in recent years, VBLinLogit remains a specialized tool for those requiring the specific advantages of VB-based ARD in a lightweight, cross-platform (MATLAB/Octave) package.

Conclusion

VBLinLogit offers a practical and efficient toolset for Bayesian linear and logistic regression. By bridging the gap between simple point-estimates and expensive MCMC methods, it enables researchers to apply sophisticated probabilistic modeling to high-dimensional datasets with minimal computational overhead.

Summary updated

BibTeX
@article{drugowitsch2019vblinlogit,
  author = {Jan Drugowitsch},
  title = {{VBLinLogit: Variational Bayesian linear and logistic regression}},
  journal = {J. Open Source Softw.},
  year = {2019},
  doi = {10.21105/joss.01359},
}
Figure