Bayesian high-dimensional semi-parametric inference beyond sub-Gaussian errors

Kyoungjae Lee, Minwoo Chae, Lizhen Lin

Abstract

We consider a sparse linear regression model with unknown symmetric error under the high-dimensional setting. The true error distribution is assumed to belong to the locally $\beta$-Hölder class with an exponentially decreasing tail, which does not need to be sub-Gaussian. We obtain posterior convergence rates of the regression coefficient and the error density, which are nearly optimal and adaptive to the unknown sparsity level. Furthermore, we derive the semi-parametric Bernstein-von Mises (BvM) theorem to characterize the asymptotic shape of the marginal posterior for regression coefficients. Under the sub-Gaussianity assumption on the true score function, strong model selection consistency for regression coefficients is also obtained, which eventually asserts the frequentist validity of credible sets.

Introduction

In high-dimensional linear regression, the objective is to estimate a coefficient vector $\theta \in \mathbb{R}^p$ where the number of predictors $p$ significantly exceeds the number of observations $n$. Traditional Bayesian approaches to this "large $p$, small $n$" problem often rely on the assumption that the error distribution is Gaussian or sub-Gaussian. However, real-world data frequently exhibits heavy tails or outliers that violate these assumptions, leading to biased estimates and unreliable uncertainty quantification.

This research addresses the challenge of providing frequentist theoretical guarantees for semi-parametric Bayesian models in high-dimensional settings without the strict sub-Gaussian error requirement. The authors establish nearly optimal posterior convergence rates and prove the semi-parametric Bernstein-von Mises (BvM) theorem, bridging the gap between flexible modeling and rigorous statistical theory.

Methodology

The authors propose a semi-parametric framework that combines sparsity-inducing priors with flexible error density estimation.

1. Sparse Prior for Coefficients

To handle high dimensionality, the model employs a spike-and-slab prior on the regression coefficients $\theta$. This prior is characterized by a mixture of a point mass at zero (the "spike") and a heavy-tailed distribution, such as a Laplace distribution (the "slab"), for the non-zero components. This structure effectively encourages sparsity while allowing for the estimation of significant effects.

2. Flexible Error Density

Instead of assuming a fixed parametric form for the error distribution, the authors use a symmetrized Dirichlet Process (DP) mixture of Gaussians. By mixing Gaussian distributions with varying locations and scales, the model can approximate a wide range of symmetric densities $\eta$. This allows the model to adapt to the unknown shape of the noise in the data.

3. Relaxed Error Assumptions

The theoretical framework moves beyond the sub-Gaussian constraint. The true error density is only required to be locally $\beta$-Hölder (a measure of smoothness) and possess exponentially decreasing tails. This accommodates a much broader class of distributions than standard Gaussian models, though it still requires the distribution to be symmetric around zero.

Theoretical Contributions

The paper provides several key mathematical proofs that validate the performance of this semi-parametric approach:

  • Adaptive Posterior Convergence: The authors prove that the posterior distribution concentrates around the true parameters at a nearly optimal rate. Specifically, the convergence rate for the regression coefficients under $\ell_1$ and $\ell_2$ norms matches minimax optimal rates up to logarithmic factors. Crucially, the model is "adaptive," meaning it achieves these rates without prior knowledge of the true sparsity level $s_n$.
  • Bernstein-von Mises (BvM) Theorem: The derivation of a semi-parametric BvM theorem is a significant milestone. It demonstrates that as $n$ grows, the marginal posterior for the regression coefficients begins to resemble a Normal distribution centered at the truth. This provides a frequentist justification for using Bayesian credible sets as valid confidence intervals.
  • Model Selection Consistency: The paper establishes that the model can correctly identify the set of non-zero predictors (strong model selection consistency) provided a "beta-min" condition is met. A key insight is that this consistency only requires the score function (the derivative of the log-likelihood) to be sub-Gaussian, rather than the error distribution itself.

Discussion and Limitations

While this work significantly advances Bayesian high-dimensional theory, some constraints remain. The current model assumes the error distribution is symmetric around zero. While symmetry is a common assumption in semi-parametric literature, it may not hold for skewed real-world data. Additionally, while the "exponentially decreasing tail" assumption is broader than sub-Gaussianity, it still excludes extremely heavy-tailed distributions like the Cauchy distribution.

Future research may explore location-scale mixtures to accommodate even heavier tails (such as Student’s $t$ distributions) and investigate methods to relax the symmetry requirement to a simpler median-zero or mean-zero constraint.

Conclusion

By relaxing the Gaussian error assumption and providing rigorous proofs for convergence and uncertainty quantification, this paper provides a robust theoretical foundation for semi-parametric Bayesian inference in high-dimensional settings. It ensures that practitioners can use flexible models for "messy" data while maintaining the mathematical guarantees required for reliable scientific inference.

Summary updated

BibTeX
@article{lee2020bayesian,
  author = {Kyoungjae Lee and Minwoo Chae and Lizhen Lin},
  title = {{Bayesian high-dimensional semi-parametric inference beyond sub-Gaussian errors}},
  journal = {Journal of the Korean Statistical Society},
  year = {2020},
  doi = {10.1007/s42952-020-00091-4},
}
Figure