Abstract: A wide range of methods, Bayesian and others, tackle regression when there are many variables. In the Bayesian context, the prior is constructed to reflect ideas of variable selection and to encourage appropriate shrinkage. The prior needs to be reasonably robust to different signal to noise structures. Two simple evergreen prior constructions stem from ridge regression on the one hand and g-priors on the other. We seek to embed recent ideas about sparsity of the regression coefficients and robustness into these priors. We also explore the gains that can be expected from these differing approaches.
Introduction
In modern data science, we frequently encounter "wide" datasets—such as those in genomics or chemometrics—where the number of predictors ($p$) often exceeds the number of observations ($n$). To prevent overfitting in these high-dimensional settings, statisticians employ priors to "shrink" unimportant coefficients toward zero.
Traditionally, two approaches have dominated: Ridge Regression, which shrinks all coefficients equally but can be overly aggressive, and g-priors, which adjust shrinkage based on variable correlations but often fail when variables are highly correlated (the "ill-conditioned" case). This paper introduces the Eigen-CNG prior, a robust Bayesian framework that bridges these two "evergreen" methods by blending the structural advantages of g-priors with the stability of ridge regression and the sparsity of normal-gamma distributions.
Methodology: The Eigen-CNG Framework
The core innovation lies in applying the Correlated Normal-Gamma (CNG) prior to the canonical reduction of the regression model using Singular Value Decomposition (SVD). By decomposing the design matrix $X$ into its singular values (eigenvalues), the authors define a prior governed by two key hyperparameters:
- $k$: Controls the relationship between the prior variance and the eigenvalues, linking the model to either ridge ($k=0$) or g-prior ($k=1$) behavior.
- $b$: A "sparsity shape parameter."
This framework allows for three special cases: Ridge-CNG (robust ridge), g-prior-CNG (robust g-prior), and Eigen-CNG, where $k$ and $b$ are learned from the data. This allows the model to automatically adapt to the underlying signal-to-noise structure. Crucially, the authors prove that as eigenvalues approach zero, their prior construction forces the distribution mass to concentrate at zero, effectively "killing off" noise in directions where the data provides no information.
Experimental Evaluation
The authors evaluated their methods on nine diverse datasets, categorized by their dimensionality:
- High-dimensional ($p > n$): Gasoline, Yarn, Cereal, N-mouse, M-drug, and Liver.
- Low-dimensional ($p < n$): CPS (wages), Crime (state crime rates), and Ozone (meteorological data).
Performance was measured using the Log Predictive Score (LPS), where lower values indicate superior predictive accuracy.
Results and Discussion
The experimental results demonstrate that the Eigen-CNG framework and its variants are highly competitive across diverse data structures.
Key Insights
- Sparsity in High Dimensions: In nearly all $p > n$ cases, sparse priors (CNG variants and the Horseshoe prior) outperformed standard ridge and g-priors.
- The Yarn Success: The Yarn dataset provided a notable victory for the Eigen-CNG approach. Standard ridge regression often fails here by over-shrinking real effects associated with small eigenvalues. The Eigen-CNG prior correctly identified these effects, leading to significantly better predictions.
- Robustness to Singularity: The model handles near-singular design matrices by making the shape parameter of the Gamma distribution vanish alongside the eigenvalue. This creates a "spiky" prior at zero that counteracts infinite variance, resulting in stable, sparse estimates.
- Low-Dimensional Performance: In $p < n$ scenarios (e.g., Crime or Ozone), classic ridge regression remains a formidable benchmark, though the CNG variants performed with comparable accuracy.
Conclusion
The Eigen-CNG framework successfully treats the choice between ridge and g-priors as a continuum, allowing the data to determine the optimal balance. While the authors note a subtle issue regarding aggregate sparsity as latent components increase, the current results show that "modestly fat" exponential tails provide an excellent balance between shrinking noise and preserving significant signals. Future work may explore more complex hierarchical structures for sparsity parameters or extend these eigen-priors to non-linear modeling contexts.
Summary updated