Safe-Bayesian Generalized Linear Regression

Rianne de Heide, Alisa Kirichenko, Peter Grunwald, Nishant A. Mehta

Abstract

We study generalized Bayesian inference under misspecification, i.e. when the model is 'wrong but useful'. Generalized Bayes equips the likelihood with a learning rate $\eta$. We show that for generalized linear models (GLMs), $\eta$-generalized Bayes concentrates around the best approximation of the truth within the model for specific $\eta \neq 1$, even under severely misspecified noise, as long as the tails of the true distribution are exponential. We derive MCMC samplers for generalized Bayesian lasso and logistic regression and give examples of both simulated and real-world data in which generalized Bayes substantially outperforms standard Bayes.

Introduction: When "Wrong but Useful" Models Fail

The core tenet of Bayesian statistics is that the model is a correct description of the data-generating process. However, as George Box famously noted, "All models are wrong." When a model is misspecified—for example, if you assume constant noise (homoscedasticity) but the real-world noise varies (heteroscedasticity)—standard Bayesian inference can overfit or even fail to converge to the best possible approximation.

Previous research showed this "bad behavior" in simple ridge regression. This paper addresses whether this failure persists in popular sparse priors like the Lasso or Horseshoe, in more complex models like Logistic Regression, and on real-world datasets.

Methods: Generalized Bayes and the Learning Rate $\eta$

The authors employ Generalized Bayesian inference (also called fractional or tempered Bayes). Instead of the standard posterior, they use a "Gibbs posterior" where the likelihood is raised to the power of a learning rate $\eta$ (usually $\eta < 1$). This effectively "downweights" the data relative to the prior, providing a stronger regularizing effect.

Key technical contributions include:

  • Theoretical Proofs: Proving that for GLMs, the "central condition" for posterior concentration holds if the true data distribution has exponential tails.
  • Novel MCMC Samplers: Deriving new algorithms, including a Polya-Gamma latent variable scheme for generalized logistic regression and a modified Gibbs sampler for the Bayesian Lasso.
  • Safe-Bayesian Algorithm: Using a heuristic to "learn" the optimal $\eta$ from the data, ensuring inference remains robust to misspecification.

Key Insights: Learning Slower to Learn Better

A common misconception is that sparsity-inducing priors like the Lasso naturally prevent overfitting under misspecification. The authors show this is false; standard Bayesian Lasso can still overfit wildly if the noise model is wrong.

Standard Bayes overfits wildly under misspecification, while Safe-Bayes remains stable.

In misspecified settings, standard Bayes tries to learn "too much" from the noise. Setting $\eta < 1$ forces the model to learn more slowly, which paradoxically leads to better long-term accuracy. This is reflected in the predictive variance: standard Bayes shows massive fluctuations in uncertainty between data points when misspecified, while Generalized Bayes maintains a stable predictive variance.

Results and Real-World Performance

In heteroscedastic regression simulations, the risk (error) for standard Bayesian Lasso and Horseshoe actually increased as more data was added initially—a phenomenon known as "bad behavior." In contrast, the Safe-Bayesian Lasso remained stable.

Risk curves showing standard methods failing to improve with more data, while Safe-Bayes stays low.

The approach also showed significant gains on real-world datasets:

  • Seattle Airport Temperatures: Safe-Bayes achieved an MSE of 6.04, outperforming Bayes Lasso (6.16) and Horseshoe (6.53).
  • London Air Pollution: Safe-Bayes achieved an MSE of 1142, compared to 1201 for Bayes Lasso and 1169 for Horseshoe.

In logistic regression experiments, standard Bayesian risk spiked to nearly 1.0, while the generalized version with $\eta = 0.125$ stayed near 0.72.

Conclusion

While standard Bayesian inference is powerful, its reliance on model correctness makes it vulnerable. Generalized Bayes with a learned learning rate $\eta$ offers a robust alternative for GLMs. Future work will focus on improving the computational efficiency of selecting $\eta$ and extending these methods to a wider variety of models.

Summary updated

Abstract

We study generalized Bayesian inference under misspecification, i.e. when the model is ‘wrong but useful’. Generalized Bayes equips the likelihood with a learning rate $\eta$. We show that for generalized linear models (GLMs), $\eta$-generalized Bayes concentrates around the best approximation of the truth within the model for specific $\eta eq 1$, even under severely misspecified noise, as long as the tails of the true distribution are exponential. We derive MCMC samplers for generalized Bayesian lasso and logistic regression and give examples of both simulated and real-world data in which generalized Bayes substantially outperforms standard Bayes.
BibTeX
@inproceedings{heide2020safe,
  author = {Rianne de Heide and Alisa Kirichenko and Peter Grunwald and Nishant A. Mehta},
  title = {{Safe-Bayesian Generalized Linear Regression}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=lOZcWinX9H},
}
Figure