Abstract
In high-dimensional data settings where $p \gg n$, many penalized regularization approaches were studied for simultaneous variable selection and estimation. However, with the existence of covariates with weak effect, many existing variable selection methods, including Lasso and its generations, cannot distinguish covariates with weak and no contribution. Thus, prediction based on a subset model of selected covariates only can be inefficient. In this paper, we propose a post selection shrinkage estimation strategy to improve the prediction performance of a selected subset model. Such a post selection shrinkage estimator (PSE) is data-adaptive and constructed by shrinking a post selection weighted ridge estimator in the direction of a selected candidate subset. Under an asymptotic distributional quadratic risk criterion, its prediction performance is explored analytically. We show that the proposed post selection PSE performs better than the post selection weighted ridge estimator. More importantly, it improves the prediction performance of any candidate subset model selected from most existing Lasso-type variable selection methods significantly. The relative performance of the post selection PSE is demonstrated by both simulation studies and real data analysis.
Introduction
In modern statistical analysis, the high-dimensional regime—where the number of features ($p$) significantly exceeds the number of observations ($n$)—has become the norm rather than the exception. To handle this complexity, researchers typically rely on parsimonious models. Methods like the Lasso (Least Absolute Shrinkage and Selection Operator) are designed to enforce sparsity, effectively assuming that only a small subset of variables truly influences the outcome.
While these methods are excellent for identifying "strong signals," they often falter in the presence of "weak signals"—covariates that have a small but non-zero impact on the response variable. Standard Lasso-type estimators frequently categorize these weak signals as noise, discarding them to maintain model simplicity. This exclusion leads to selection bias and a loss of predictive power. This paper introduces the Post Selection Shrinkage Estimator (PSE), a novel strategy designed to bridge the gap between sparse selection and predictive robustness by recapturing the information contained in these omitted weak signals.
The Problem: The Hidden Cost of Sparsity
The fundamental challenge addressed here is the "Three-Category Universe" of variables:
- Strong Signals ($S_1$): Variables with large effects that are easily identified by selection methods.
- Weak Signals ($S_2$): Variables with small effects that are often discarded by Lasso but collectively contribute to prediction.
- Zero Effects ($S_3$): Pure noise variables with no relationship to the outcome.
Existing high-dimensional theory often collapses $S_1$ and $S_2$ into a single "active" set, but in practice, Lasso-type methods are only efficient at finding $S_1$. When $S_2$ is ignored, the resulting model is under-fitted, leading to suboptimal prediction performance. The PSE aims to protect against this under-fitting by utilizing a shrinkage mechanism that accounts for the joint contribution of the $S_2$ set.
Methodology: The Post Selection Shrinkage Estimator (PSE)
The authors propose a three-step data-adaptive strategy to construct the PSE:
1. Initial Selection
First, a standard variable selection method (such as Lasso or Adaptive Lasso) is applied to identify a candidate subset of strong variables ($S_1$). This provides a baseline sparse model that is interpretable but potentially biased.
2. Weighted Ridge Estimation
Next, the method constructs a "weighted ridge" estimator. Unlike a standard ridge regression that shrinks all coefficients equally, this version applies a thresholding penalty. This allows the model to continue ignoring pure noise ($S_3$) while retaining the joint contribution of the weak signals ($S_2$) that the initial selection step missed. This estimator serves as a "fuller" model that contains more information but may suffer from higher variance.
3. Stein-type Shrinkage
The final step is the application of a Stein-type shrinkage rule. This rule mathematically blends the sparse model from Step 1 with the weighted ridge model from Step 2. The PSE is inherently data-adaptive: if the sparse model performs well, the estimator remains close to the Lasso-selected subset; if the omitted variables contain significant predictive power, the PSE pulls in information from the weighted ridge estimator to compensate.
Empirical Evaluation
Simulation Studies
The authors conducted extensive simulations across various configurations of strong, weak, and zero signals. They specifically examined how the model behaves as the dimensionality ($p$) grows relative to the sample size ($n$). The results consistently showed that the PSE achieved lower Asymptotic Distributional Risk (ADR) than either the post-selection least squares or the weighted ridge estimators alone. Notably, as the $p \gg n$ regime became more extreme, the advantage of the PSE over standard Lasso became more pronounced, demonstrating its robustness in high-dimensional settings.
Real-World Application: Economic Growth Data
To test the PSE in a practical context, the authors analyzed a classic economic dataset from Barro & Lee (1994). The goal was to predict annualized GDP growth rates (1960–1985) across 139 countries using 90 predictors (including interactions) for only 82 observations.
In this $p > n$ scenario, the PSE significantly outperformed the Adaptive Lasso. Using the Relative Residual Sum of Squares (RRSS) as a metric, the PSE demonstrated superior predictive accuracy across various threshold settings, proving that the "weak signals" in economic indicators—which are often individually insignificant—are collectively vital for accurate forecasting.
Conclusion
The Post Selection Shrinkage Estimator (PSE) offers a powerful refinement to the standard high-dimensional toolkit. By acknowledging that weak signals are distinct from noise, the PSE provides a "protection layer" against the selection bias inherent in sparse models. While the method requires careful tuning of parameters via cross-validation, its ability to improve the prediction performance of any candidate subset model makes it a valuable contribution to the field of high-dimensional data analysis. Future research may explore the use of alternative base estimators to further enhance the flexibility of this shrinkage strategy.
Summary updated