Numerical Characterization of Support Recovery in Sparse Regression with Correlated Design

Ankit Kumar, Sharmodeep Bhattacharyya, Kristofer E. BouchardCommun. Stat. Simul. Comput 2021

TL;DR

This paper investigates how well different sparse regression methods can identify the true underlying features when those features are correlated. They find a fundamental trade-off between false positives and false negatives, and identify specific combinations of estimators and model selection criteria (SCAD with BIC or empirical Bayes) that perform best across various scenarios.

What Problem Does This Paper Address?

Sparse regression is a powerful tool for feature selection in many scientific fields. However, real-world data often features correlated predictors, which can significantly hinder the ability of these methods to accurately identify the true set of important features (support recovery) and estimate their coefficients. While theoretical work has explored conditions for consistent selection, a systematic numerical study of how different estimators and model selection strategies perform in the presence of correlated designs, across a range of signal-to-noise ratios and model densities, has been lacking. This paper aims to fill that gap by providing a comprehensive numerical characterization of this problem.

Methods

The authors conduct an extensive simulation study. They evaluate five different sparse regression estimators: Lasso, Elastic Net, SCAD, MCP, and a novel UoI Lasso. For model selection, they consider five criteria: Cross-Validation (CV/R²), BIC, AIC, gMDL, and Empirical Bayes. The core of their methodology involves:

  • Systematic Simulation: Generating a wide variety of synthetic datasets with varying degrees of feature correlation, signal-to-noise ratios (SNR), sample sizes (n/p ratios), and true model densities (sparsity levels).
  • Performance Metrics: Evaluating performance using selection accuracy, false negative rate (FNR), and false positive rate (FPR).
  • Parameter Exploration: Analyzing the impact of a parameter 'α', which quantifies the difficulty of the regression problem due to correlation and signal strength.
  • Comparison of Combinations: Assessing the performance of each estimator-selection criterion pair (referred to as "inference algorithms").

Datasets

Not discussed in the paper. The study relies entirely on synthetic data generated according to various parameters.

Key Insights and Ideas

  • Fundamental Trade-off: All tested estimators and selection methods exhibit a fundamental trade-off between controlling false positives and false negatives. The optimal balance depends on the specific problem characteristics.
  • Universality of 'α': The parameter 'α', derived from theoretical bounds on optimal recovery, proves to be a useful and surprisingly universal measure of problem difficulty, modulating selection accuracy across different estimators and methods.
  • Transition Points: There are identifiable "transition points" in 'α' beyond which selection accuracy degrades significantly, and false negative/positive magnitudes inflate. This suggests a potential heuristic for practitioners to assess problem difficulty.
  • Estimator vs. Selector: Both the choice of the sparse regression estimator and the model selection criterion are crucial for good performance. They interact in complex ways.
  • SCAD/BIC/Empirical Bayes Shine: SCAD, when paired with BIC or empirical Bayes model selection, consistently demonstrates robust feature selection performance across a wide range of correlated regression problems.

Key Results

  • Best Performing Combination: SCAD with BIC or empirical Bayes model selection is identified as the most robust combination for feature selection across the tested scenarios.
  • False Positive/Negative Behavior:
    • The false positive rate is largely insensitive to correlation strength ('α') for most estimators, but is strongly dependent on the estimator itself (SCAD/MCP/UoI generally have lower FPR than Lasso/Elastic Net).
  • The false negative rate shows a more pronounced dependence on 'α', with a clear transition point where it increases rapidly as 'α' decreases (indicating higher correlation/lower signal).
  • Model Density Impact: Denser models (higher k/p) require more careful control of false negatives, while sparser models demand better false positive control.
  • Magnitude of Errors: The distortion in coefficient estimates (magnitude of false positives and negatives) also shows universal transition points related to 'α', becoming more severe in denser models.
  • Cross-Validation and AIC Issues: Cross-validation and AIC tend to lead to higher false positive rates, especially in sparser models, often resulting in lower selection accuracy compared to BIC.

Limitations and Future Work

  • Theoretical Gaps: The authors note that theoretical results for non-asymptotic behavior of regularization strength selection are still lacking for many criteria.
  • Adaptive Penalties: The empirical Bayes and gMDL methods, while showing promise, did not consistently outperform BIC across all conditions, suggesting room for methodological development in adaptive complexity penalties.
  • Universal Transition Point: The observed universal 'α'-transition point for false negatives and coefficient distortion is highlighted as a novel finding, with potential for future theoretical analysis.
  • Average Case Analysis: Future work could involve an average-case analysis of coefficient support distortion as a function of 'α' and spectral properties of the design matrix.
  • Nonconvex Optimization: While SCAD and MCP are nonconvex, recent work suggests stationary points have similar performance, and optimization algorithms are maturing, making them more practical.

Summary updated

Abstract

Sparse regression is frequently employed in diverse scientific settings as a feature selection method. A pervasive aspect of scientific data that hampers both feature selection and estimation is the presence of strong correlations between predictive features. These fundamental issues are often not appreciated by practitioners, and jeapordize conclusions drawn from estimated models. On the other hand, theoretical results on sparsity-inducing regularized regression such as the Lasso have largely addressed conditions for selection consistency via asymptotics, and disregard the problem of model selection, whereby regularization parameters are chosen. In this numerical study, we address these issues through exhaustive characterization of the performance of several regression estimators, coupled with a range of model selection strategies. These estimators and selection criteria were examined across correlated regression problems with varying degrees of signal to noise, distribution of the non-zero model coefficients, and model sparsity. Our results reveal a fundamental tradeoff between false positive and false negative control in all regression estimators and model selection criteria examined. Additionally, we are able to numerically explore a transition point modulated by the signal-to-noise ratio and spectral properties of the design covariance matrix at which the selection accuracy of all considered algorithms degrades. Overall, we find that SCAD coupled with BIC or empirical Bayes model selection performs the best feature selection across the regression problems considered.
BibTeX
@article{kumar2021numerical,
  author = {Ankit Kumar and Sharmodeep Bhattacharyya and Kristofer E. Bouchard},
  title = {{Numerical Characterization of Support Recovery in Sparse Regression with Correlated Design}},
  journal = {Commun. Stat. Simul. Comput.},
  year = {2021},
}
Figure