Tail-adaptive Bayesian shrinkage

Se Yoon Lee, Bani Mallick, Debdeep PatiElectronic Journal of Statistics 2026 · OpenReview

TL;DR

This paper introduces a novel "tail-adaptive" Bayesian shrinkage prior (GLT prior) for high-dimensional regression. Unlike existing methods that struggle with moderate sparsity, the GLT prior dynamically adjusts its "tail-heaviness" to better capture signals across various sparsity levels, leading to more robust and accurate estimations.

What Problem Does This Paper Address?

In high-dimensional settings, we often have many more predictors (features) than observations. A common goal is to identify a small subset of "important" predictors that influence the outcome variable. This is known as the "sparse" setting. Traditional Bayesian shrinkage methods, like the Horseshoe prior, are excellent at identifying a very small number of signals (ultra-sparsity). However, they tend to perform poorly when the number of signals is moderate, leading to a "collapsing behavior" where the method incorrectly concludes there are no signals. This paper addresses the challenge of developing a Bayesian shrinkage method that remains effective and robust across a wide range of sparsity levels, from ultra-sparse to moderately sparse.

Methods

The core contribution is the Global-Local-Tail (GLT) Gaussian mixture distribution. This new prior family extends existing global-local shrinkage priors by introducing a shape parameter ($\xi$) that is treated as a random variable. This allows the prior's tail-heaviness to adapt dynamically:

  • Ultra-sparse: The prior becomes "thinner" to focus on a few signals.
  • Moderately sparse: The prior becomes "heavier" to accommodate more signals.

This "tail-adaptive" property is crucial for robustness. The authors develop a Markov Chain Monte Carlo (MCMC) sampling algorithm, incorporating an elliptical slice sampler and a Hill estimator, to efficiently estimate the parameters, including the shape parameter $\xi$. They also provide theoretical guarantees on the GLT posterior's convergence properties, showing it achieves near-minimax optimal rates.

Datasets

  • Prostate Cancer Data: A real-world dataset used to demonstrate the GLT prior's performance on gene expression data.
  • Simulated Datasets: The paper extensively uses simulated data to evaluate the GLT prior and compare it against the Horseshoe prior under various scenarios:
  • Varied sparsity levels (from ultra-sparse to moderate).
  • Varied degrees of multicollinearity (correlation between predictors).
  • Varied signal-to-noise ratios.

Key Insights and Ideas

  • The "Collapsing Behavior" of the Horseshoe: The paper highlights that the Horseshoe prior, while excellent for ultra-sparsity, suffers from a fixed tail-heaviness. This inflexibility leads to a "collapsing behavior" where it incorrectly identifies no signals when sparsity is moderate.
  • Tail-Adaptivity is Key: The GLT prior's ability to adapt its tail-heaviness based on the estimated sparsity level is its main advantage. This dynamic adjustment allows it to perform well across a much wider range of sparsity scenarios.
  • The Shape Parameter ($\xi$) as a Control: The random shape parameter $\xi$ acts as a crucial knob, allowing the prior to adjust its tail behavior. Estimating this parameter effectively is a significant part of the GLT framework.
  • Theoretical Guarantees: The GLT posterior not only performs well empirically but also has strong theoretical backing, contracting at optimal rates, which is essential for reliable inference.

Key Results

  • Robustness Across Sparsity Levels: Simulations show the GLT prior significantly outperforms the Horseshoe in terms of Mean Squared Error (MSE) when sparsity levels increase beyond a certain threshold (around 0.022).
  • Improved Performance with Multicollinearity and Low SNR: The GLT prior also demonstrates superior robustness compared to the Horseshoe in scenarios with high multicollinearity and low signal-to-noise ratios.
  • Prostate Cancer Data Analysis: The GLT prior successfully identified signals in the prostate cancer dataset, maintaining its performance even with a large number of genes, whereas the Horseshoe showed signs of collapsing.
  • Theoretical Convergence: The GLT posterior is shown to contract at near-minimax optimal rates, providing strong theoretical justification for its use.

Limitations and Future Work

  • Delicate Care in Estimating $\xi$: The authors acknowledge that estimating the shape parameter $\xi$ requires careful handling, which is why they developed a specialized MCMC algorithm.
  • Future Work: The paper mentions that the Supplementary Material includes further exploration of the GLT prior's performance compared to other shrinkage priors and its application to curve fitting studies. The current work focuses on linear regression, and extensions to other model types could be a future direction.

Summary updated

Abstract

Robust Bayesian methods for high-dimensional regression problems under diverse sparse regimes are studied. Traditional shrinkage priors are primarily designed to detect a handful of signals from tens of thousands of predictors in the so-called ultra-sparsity domain. However, they may not perform desirably when the degree of sparsity is moderate. In this paper, we propose a robust sparse estimation method under diverse sparsity regimes, which has a tail-adaptive shrinkage property. In this property, the tail-heaviness of the prior adjusts adaptively, becoming larger or smaller as the sparsity level increases or decreases, respectively, to accommodate more or fewer signals, a posteriori. We propose a global-local-tail (GLT) Gaussian mixture distribution that ensures this property. We examine the role of the tail-index of the prior in relation to the underlying sparsity level and demonstrate that the GLT posterior contracts at the minimax optimal rate for sparse normal mean models. We apply both the GLT prior and the Horseshoe prior to a real data problem and simulation examples. Our findings indicate that the varying tail rule based on the GLT prior offers advantages over a fixed tail rule based on the Horseshoe prior in diverse sparsity regimes.
BibTeX
@article{lee2026tail,
  author = {Se Yoon Lee and Bani Mallick and Debdeep Pati},
  title = {{Tail-adaptive Bayesian shrinkage}},
  journal = {Electronic Journal of Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=kOBZKKcHQj},
}
Figure