Abstract
We propose a new method, semi-penalized inference with direct false discovery rate control (SPIDR), for variable selection and confidence interval construction in high-dimensional linear regression. SPIDR first uses a semi-penalized approach to constructing estimators of the regression coefficients. We show that the SPIDR estimator is ideal in the sense that it equals an ideal least squares estimator with high probability under a sparsity and other suitable conditions. Consequently, the SPIDR estimator is asymptotically normal. Based on this distributional result, SPIDR determines the selection rule by directly controlling false discovery rate. This provides an explicit assessment of the selection error. This also naturally leads to confidence intervals for the selected coefficients with a proper confidence statement. We conduct simulation studies to evaluate its finite sample performance and demonstrate its application on a breast cancer gene expression data set. Our simulation studies and data example suggest that SPIDR is a useful method for high-dimensional statistical inference in practice.
Introduction
In modern high-dimensional data analysis, such as genomics and precision medicine, researchers often face the challenge of linear regression where the number of predictors ($p$) far exceeds the number of samples ($n$). While popular penalized methods like the Lasso or MCP (Minimax Concave Penalty) are effective for variable selection, they typically lack a straightforward framework for uncertainty quantification. Specifically, providing reliable p-values or confidence intervals for selected variables remains difficult. Furthermore, these traditional methods often struggle when predictors are highly correlated—a common scenario in biological pathways—leading to unstable selections and missed signals.
The SPIDR Methodology
To address these limitations, we introduce SPIDR (Semi-Penalized Inference with direct false Discovery Rate control). The core of this method is a "one-at-a-time" semi-penalized estimation approach. Instead of penalizing all coefficients simultaneously, SPIDR focuses on one coefficient ($\beta_j$) at a time. For each variable, it solves an optimization problem where $\beta_j$ is unpenalized, while all other variables are penalized using the MCP.
This formulation allows the SPIDR estimator to achieve "idealness," behaving like a standard least-squares estimator as if the true model support were known. Theoretically, we prove that the SPIDR estimator is asymptotically normal. This distributional result is pivotal, as it allows for the direct application of False Discovery Rate (FDR) control methods (e.g., Storey 2002). Researchers can thus specify a concrete "error budget" (e.g., a 10% FDR) and obtain valid, FDR-adjusted confidence intervals for every selected variable.
Key Insights: Stableness and Stickiness
A significant advantage of SPIDR is its stableness and stickiness. In traditional penalized regression, the set of selected variables can fluctuate wildly with small changes in the tuning parameter $\lambda$. SPIDR’s solution paths, however, are remarkably stable. "Stickiness" refers to the method's ability to capture strong signals even when they are highly correlated with other predictors, as its selection rule depends on residual signal strength rather than simple pairwise correlation.
As shown in the figure above, the solution paths for SPIDR (columns 3-5) remain relatively constant across a wide range of $\log(\lambda)$ compared to the shrinking paths of Lasso and MCP. This stability is what facilitates reliable inference in high-dimensional settings.
Results and Applications
Simulation Studies
In synthetic models with $p=1000$ and $n \approx 160$, SPIDR demonstrated superior performance in handling "messy" data. In scenarios with high pairwise correlation ($r=0.8$), the Lasso missed 12 out of 18 true signals. In contrast, SPIDR maintained a low FDR and a significantly lower False Miss Rate (FMR) than both Lasso and MCP.
Breast Cancer Gene Expression Data
We applied SPIDR to real-world data from The Cancer Genome Atlas (TCGA), modeling the expression of the BRCA1 gene using 1,685 candidate genes from 519 patients. SPIDR identified 63 significant genes at a 10% FDR level. Notably, it successfully identified genes like UHRF1 and SRPK1—known in the literature to be involved in cancer progression and BRCA1 repression—which were completely missed by both Lasso and MCP.
Conclusion
SPIDR provides a robust framework for high-dimensional statistical inference by bridging the gap between penalized estimation and large-scale hypothesis testing. While more computationally intensive than a single Lasso run, its ability to provide valid confidence intervals and handle high correlation makes it a valuable tool for modern data science. Future work will explore extending this framework to Generalized Linear Models and survival analysis.
Summary updated