Abstract
In this paper, we develop a new estimation and valid inference method for single or low-dimensional regression coefficients in high-dimensional generalized linear models. The number of the predictors is allowed to grow exponentially fast with respect to the sample size. The proposed estimator is computed by solving a score function. We recursively conduct model selection to reduce the dimensionality from high to a moderate scale and construct the score equation based on the selected variables. The proposed confidence interval (CI) achieves valid coverage without assuming consistency of the model selection procedure. When the selection consistency is achieved, we show the length of the proposed CI is asymptotically the same as the CI of the "oracle" method which works as well as if the support of the control variables were known. In addition, we prove the proposed CI is asymptotically narrower than the CIs constructed based on the de-sparsified Lasso estimator (van de Geer et al., 2014) and the decorrelated score statistic (Ning and Liu, 2017). Simulation studies and real data applications are presented to back up our theoretical findings.
Introduction
In modern data science, researchers frequently encounter "high-dimensional" problems where the number of potential predictors ($p$) far exceeds the number of observations ($n$). For instance, in genomics, one might measure thousands of gene expressions across only a few dozen patients. A central challenge in this setting is performing valid statistical inference—specifically, constructing reliable confidence intervals (CIs) for the effect of a specific variable of interest.
Standard statistical tests often fail in high dimensions because the process of selecting relevant variables (model selection) introduces bias. While existing solutions like the "de-sparsified Lasso" attempt to correct this bias, they often result in excessively wide confidence intervals. In practical terms, a wide CI is like a weather forecast predicting a temperature between -20°C and +40°C; while technically accurate, it lacks the precision necessary for informed decision-making. This paper introduces the Recursive Online-Score Estimation (ROSE) method to bridge this gap, providing an estimator that achieves "oracle" efficiency—performing as well as if the true underlying model were known from the start.
Methodology: The ROSE Framework
ROSE departs from traditional "one-shot" de-biasing methods by adopting a recursive approach inspired by online learning. The procedure follows four key steps:
- Data Chunking: The dataset is divided into a series of non-overlapping "chunks."
- Recursive Selection: The algorithm uses the first chunk to select a set of relevant "control" variables. It then uses the subsequent chunk to evaluate a score function based on those selected variables.
- Iterative Refinement: This process repeats. The first $k$ chunks are used for selection, and the $(k+1)$-th chunk is used for evaluation. This ensures that the data used to "pick" the model is independent of the data used to "test" the model at each step, mitigating selection bias.
- Recursive Score Equation: The final estimate is obtained by solving a recursive score equation. By reducing dimensionality from a high to a moderate scale during recursion, ROSE avoids the heavy variance penalty typically associated with traditional de-biasing techniques.
This framework is applicable to Generalized Linear Models (GLMs), including linear regression for continuous outcomes and logistic regression for binary outcomes.
Key Insights and Theoretical Advantages
Efficiency and the Oracle Property
Traditional methods often pay a "variance tax" to remove bias. ROSE handles the bias-variance tradeoff more gracefully. Under certain conditions, ROSE achieves the "oracle" property, meaning its confidence intervals are as narrow as those produced by a researcher who already knew exactly which control variables were important.
Validity without Selection Consistency
Remarkably, the confidence intervals generated by ROSE remain valid even if the model selection step is not perfectly consistent (i.e., if it occasionally fails to pick the exact set of true predictors). While it performs best when selection is accurate, this robustness is a significant theoretical advantage over methods that rely strictly on perfect selection.
Superior Precision
The authors mathematically prove that ROSE's confidence intervals are asymptotically narrower than those produced by the de-sparsified Lasso or the decorrelated score statistic. This increased precision allows researchers to detect subtle signals that other methods might overlook.
Empirical Evaluation
Simulation Studies
The authors tested ROSE in various high-dimensional scenarios (e.g., $p=1000, n=100$ or $200$). In nearly all cases, ROSE maintained the target 95% coverage rate while consistently producing shorter intervals than competing methods.
Real-World Application: Riboflavin Production
The method was applied to a classic benchmark dataset involving vitamin B2 production in Bacillus subtilis. This dataset is extremely high-dimensional, containing only 71 observations but 4,088 gene expression predictors.
- De-sparsified Lasso: Failed to identify any significant genes after multiple comparison adjustments.
- ROSE: Successfully identified three significant genes (the 1588th, 3154th, and 4004th).
This result demonstrates that ROSE possesses the statistical power to find "needles in the haystack" where traditional high-dimensional inference methods see only noise.
Limitations and Future Directions
While powerful, ROSE has specific requirements and costs:
- Signal Strength: The procedure relies on the important variables being "loud" enough to be selected during the recursive steps. A "doubly-robust" variant is proposed for cases with weaker signals.
- Computational Complexity: Re-estimating the model for every chunk is more intensive than one-shot methods. The authors suggest using larger chunks to improve computational efficiency.
- Multivariate Extensions: While the current work focuses on single parameters, the framework can be extended to test multiple variables simultaneously, representing a promising area for future empirical study.
Summary updated