Abstract
High-dimensional regression typically relies on complex penalization methods like the Lasso, which often require restrictive conditions on variable correlation and noise distribution. This paper introduces a generalized Ordinary Least Squares (OLS) approach combined with adaptive hard thresholding. The proposed Least-squares Adaptive Thresholding (LAT) and Ridge Adaptive Thresholding (RAT) algorithms are computationally efficient, robust to heavy-tailed noise, and outperform standard penalized methods in both synthetic and real-world high-dimensional settings.
Introduction
In modern data science, high-dimensional problems where the number of predictors $p$ exceeds the number of observations $n$ are ubiquitous. Standard OLS is traditionally inapplicable in these scenarios because the design matrix $X^T X$ is singular. Conventional solutions involve penalization (e.g., Lasso, SCAD), which imposes sparsity by adding a penalty term to the objective function. However, these methods suffer from "tears": they require the "irrepresentable condition" for variable selection consistency and often assume sub-Gaussian noise. This work demonstrates that a simple, thresholded OLS estimator can overcome these limitations.
Methods: LAT and RAT Algorithms
The authors propose a three-stage framework to leverage OLS in high dimensions:
- Pre-selection Stage: A high-dimensional OLS estimator is defined as the limit of ridge regression as the penalty approaches zero. This estimator ranks all $p$ variables, allowing for the selection of a manageable submodel of size $d < n$.
- Hard Thresholding Stage: Standard OLS is applied to the submodel. A "hard threshold" filter is then used to eliminate variables whose coefficients do not significantly exceed the estimated noise level.
- Refinement Stage: A final OLS fit is performed on the remaining variables to produce unbiased coefficient estimates.
The Ridge Adaptive Thresholding (RAT) variant incorporates a small ridge penalty during intermediate steps to enhance numerical stability in the presence of high variable correlation.
Key Insights: Diagonal Dominance
A fundamental discovery of this research is the "diagonal dominance" of the high-dimensional OLS projection matrix $X^T(XX^T)^{-1}X$. Even when $p \gg n$, the diagonal elements of this matrix tend to be significantly larger than the off-diagonal elements across various correlation structures. This property allows the estimator to effectively distinguish signal from noise based on coefficient magnitudes.
Results
The performance of LAT and RAT was evaluated against synthetic benchmarks and real-world data.
Synthetic Scenarios
In simulations involving independent predictors, compound symmetry, group structures, and factor models, LAT and RAT consistently achieved lower Root Mean Squared Error (RMSE) and superior support recovery compared to Lasso. Notably, the methods remained robust under heavy-tailed noise distributions where Lasso's performance degraded.
Real-World Application: Student Performance
On the Student Performance Dataset (expanded to 767 features), the RAT algorithm achieved the lowest cross-validation error among all tested methods, including Lasso, Elastic Net, and Adaptive Lasso.
Conclusion
By returning to the simplicity of OLS, the LAT and RAT algorithms provide a powerful alternative to penalized regression. They offer significant computational speedups—being embarrassingly parallelizable in the pre-selection stage—and greater robustness to data irregularities. Future work will explore extending this framework to Generalized Linear Models and survival analysis.
Summary updated