TL;DR
This paper investigates how well different sparse regression methods can identify the true underlying features when those features are correlated. They find a fundamental trade-off between false positives and false negatives, and identify specific combinations of estimators and model selection criteria (SCAD with BIC or empirical Bayes) that perform best across various scenarios.
What Problem Does This Paper Address?
Sparse regression is a powerful tool for feature selection in many scientific fields. However, real-world data often features correlated predictors, which can significantly hinder the ability of these methods to accurately identify the true set of important features (support recovery) and estimate their coefficients. While theoretical work has explored conditions for consistent selection, a systematic numerical study of how different estimators and model selection strategies perform in the presence of correlated designs, across a range of signal-to-noise ratios and model densities, has been lacking. This paper aims to fill that gap by providing a comprehensive numerical characterization of this problem.
Methods
The authors conduct an extensive simulation study. They evaluate five different sparse regression estimators: Lasso, Elastic Net, SCAD, MCP, and a novel UoI Lasso. For model selection, they consider five criteria: Cross-Validation (CV/R²), BIC, AIC, gMDL, and Empirical Bayes. The core of their methodology involves:
- Systematic Simulation: Generating a wide variety of synthetic datasets with varying degrees of feature correlation, signal-to-noise ratios (SNR), sample sizes (n/p ratios), and true model densities (sparsity levels).
- Performance Metrics: Evaluating performance using selection accuracy, false negative rate (FNR), and false positive rate (FPR).
- Parameter Exploration: Analyzing the impact of a parameter 'α', which quantifies the difficulty of the regression problem due to correlation and signal strength.
- Comparison of Combinations: Assessing the performance of each estimator-selection criterion pair (referred to as "inference algorithms").
Datasets
Not discussed in the paper. The study relies entirely on synthetic data generated according to various parameters.
Key Insights and Ideas
- Fundamental Trade-off: All tested estimators and selection methods exhibit a fundamental trade-off between controlling false positives and false negatives. The optimal balance depends on the specific problem characteristics.
- Universality of 'α': The parameter 'α', derived from theoretical bounds on optimal recovery, proves to be a useful and surprisingly universal measure of problem difficulty, modulating selection accuracy across different estimators and methods.
- Transition Points: There are identifiable "transition points" in 'α' beyond which selection accuracy degrades significantly, and false negative/positive magnitudes inflate. This suggests a potential heuristic for practitioners to assess problem difficulty.
- Estimator vs. Selector: Both the choice of the sparse regression estimator and the model selection criterion are crucial for good performance. They interact in complex ways.
- SCAD/BIC/Empirical Bayes Shine: SCAD, when paired with BIC or empirical Bayes model selection, consistently demonstrates robust feature selection performance across a wide range of correlated regression problems.
Key Results
- Best Performing Combination: SCAD with BIC or empirical Bayes model selection is identified as the most robust combination for feature selection across the tested scenarios.
- False Positive/Negative Behavior:
- The false positive rate is largely insensitive to correlation strength ('α') for most estimators, but is strongly dependent on the estimator itself (SCAD/MCP/UoI generally have lower FPR than Lasso/Elastic Net).
- The false negative rate shows a more pronounced dependence on 'α', with a clear transition point where it increases rapidly as 'α' decreases (indicating higher correlation/lower signal).
- Model Density Impact: Denser models (higher k/p) require more careful control of false negatives, while sparser models demand better false positive control.
- Magnitude of Errors: The distortion in coefficient estimates (magnitude of false positives and negatives) also shows universal transition points related to 'α', becoming more severe in denser models.
- Cross-Validation and AIC Issues: Cross-validation and AIC tend to lead to higher false positive rates, especially in sparser models, often resulting in lower selection accuracy compared to BIC.
Limitations and Future Work
- Theoretical Gaps: The authors note that theoretical results for non-asymptotic behavior of regularization strength selection are still lacking for many criteria.
- Adaptive Penalties: The empirical Bayes and gMDL methods, while showing promise, did not consistently outperform BIC across all conditions, suggesting room for methodological development in adaptive complexity penalties.
- Universal Transition Point: The observed universal 'α'-transition point for false negatives and coefficient distortion is highlighted as a novel finding, with potential for future theoretical analysis.
- Average Case Analysis: Future work could involve an average-case analysis of coefficient support distortion as a function of 'α' and spectral properties of the design matrix.
- Nonconvex Optimization: While SCAD and MCP are nonconvex, recent work suggests stationary points have similar performance, and optimization algorithms are maturing, making them more practical.
Summary updated