Abstract
This paper considers a Bayesian analysis of the linear regression model under independent sampling from general scale mixtures of Normals. Using a common reference prior, we investigate the validity of Bayesian inference and the existence of posterior moments of the regression and scale parameters. We find that whereas existence of the posterior distribution does not depend on the choice of the design matrix or the mixing distribution, both of them can crucially intervene in the existence of posterior moments. We identify some useful characteristics that allow for an easy verification of the existence of a wide range of moments. In addition, we provide full characterizations under sampling from finite mixtures of Normals, Pearson VII or certain Modulated Normal distributions. For empirical applications, a numerical implementation based on the Gibbs sampler is recommended.
Introduction
Standard linear regression models typically assume that error terms follow a Normal distribution. However, empirical data frequently exhibits "heavy tails" or outliers that the Normal distribution fails to capture adequately. To address this, statisticians employ scale mixtures of Normals (SMN). This broad family of distributions—which includes the Student-$t$, Laplace, and contaminated Normal distributions—is constructed by treating the variance of a Normal distribution as a random variable governed by a mixing distribution.
In Bayesian analysis, researchers often use non-informative "improper" priors to reflect minimal prior knowledge. However, improper priors carry the risk of producing improper posteriors, which are mathematically invalid for inference. A significant danger in modern computational statistics is that Markov chain Monte Carlo (MCMC) methods may produce numerical results even when the underlying posterior does not exist. This paper establishes the exact theoretical conditions under which the posterior distribution and its moments (such as mean and variance) are well-defined when using SMN errors and improper priors.
Methodology and the SMN Framework
The study utilizes mathematical analysis to bound the integrals of posterior density. The authors categorize a wide array of SMN distributions and their corresponding mixing distributions, as summarized in the following table:
To formalize the conditions for posterior propriety, the paper introduces two diagnostic concepts:
- The Singularity Index ($p_j$) of the design matrix: This measures the geometric structure of the data, specifically how close subsets of explanatory variables are to being collinear.
- The Moment Index ($m$) of the mixing distribution: This characterizes the tail behavior of the specific SMN distribution chosen for the errors.
Key Theoretical Insights
Posterior Propriety vs. Moment Fragility
A central finding of the paper is that the existence of a valid posterior distribution is remarkably robust. The authors prove that a proper posterior exists if and only if the sample size ($n$) exceeds the number of regressors ($k$) by at least one ($n \ge k + 1$). This condition holds regardless of the specific heavy-tailed distribution or the geometry of the design matrix.
However, the existence of posterior moments (e.g., the expected value of regression coefficients) is far more fragile. The ability to calculate a valid mean or variance depends on the interaction between the data's singularity index and the error distribution's moment index. Specifically, posterior moments of order $r$ (where $r=1$ is the mean) never exist if $r \ge n - k$. Thus, the degrees of freedom in the model impose an absolute ceiling on the moments that can be estimated.
Distribution-Specific Characterizations
The paper provides detailed conditions for various distributions:
- Finite mixtures of Normals: Moments exist for any $r < n - k$.
- Pearson VII / Student-$t$: The condition is stricter: moments exist if $r < \min\{n-k, n-k-p_j+\nu\}$, where $\nu$ represents the degrees of freedom. This demonstrates how both heavy tails (low $ u$) and poor data geometry (high $p_j$) can invalidate posterior inferences.
Practical Implications
The research serves as a critical warning for applied statisticians: the convergence of a Gibbs sampler does not guarantee the existence of the posterior mean. If theoretical conditions for moment existence are not met, MCMC sample averages will fail to converge to a true value. For implementation, the authors recommend Gibbs sampling over Independent Monte Carlo, as the conditional Normality of SMN models allows for highly efficient updates.
Limitations and Future Work
The current analysis assumes a known mixing distribution (e.g., fixed degrees of freedom for a Student-$t$ model). Future research is needed to address scenarios where the parameters of the mixing distribution are unknown and must be estimated from the data. Additionally, exploring models with non-independent error terms remains an open area for investigation.
Summary updated