A power analysis for model-X knockoffs with ℓp-regularized statistics

Asaf Weinstein, Weijie Su, Małgorzata Bogdan, Rina Foygel Barber, Emmanuel J. CandèsThe Annals of Statistics 2023 · DOI

TL;DR

This paper develops a theoretical framework to analyze the power of variable selection procedures that use "Model-X knockoffs" to control false discovery rates in high-dimensional linear regression. The authors extend approximate message-passing (AMP) theory to derive exact asymptotic predictions for power, demonstrating that knockoff-based methods, particularly thresholded-Lasso, can achieve superior performance compared to traditional Lasso, especially when tuning is done via cross-validation.

What Problem Does This Paper Address?

In high-dimensional linear regression (where the number of predictors $p$ is much larger than the number of samples $n$), identifying the truly important predictors is a crucial task. While methods like Lasso are popular for their ability to produce sparse solutions, they often struggle to achieve perfect variable selection, leading to both false positives (Type I errors) and false negatives. Existing theoretical work has focused on predicting the trade-off between false discovery proportion (FDP) and true positive proportion (TPP) for penalized estimators. However, in practice, controlling the Type I error rate (like the False Discovery Rate, FDR) is essential, and this requires a way to calibrate these procedures without prior knowledge of the signal or noise. This paper addresses the challenge of providing a theoretically grounded analysis of the power (TPP) of selection procedures that use Model-X knockoffs to control FDR, particularly for ℓp-regularized estimators, and compares their performance to oracle procedures.

Methods

The core of this work lies in extending the theory of Approximate Message Passing (AMP). The authors leverage AMP to derive exact asymptotic predictions for the FDP and TPP of selection procedures. A key technical contribution is the extension of existing AMP convergence results to handle the joint distribution of original variables and their knockoff counterparts, which is crucial for analyzing knockoff-based methods. They specifically analyze the performance of Lasso and thresholded-Lasso selection procedures when augmented with Model-X knockoffs. The paper also investigates the optimal tuning parameter selection for these procedures, proposing and analyzing a cross-validation approach on the augmented design matrix. Furthermore, they extend their analysis to Type S errors (errors in determining the sign of a coefficient), providing predictions for false sign proportion (FSP) and a corresponding notion of power.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and simulations based on synthetic data generated according to the described linear model.

Key Insights and Ideas

  • Knockoffs Enhance Power: Model-X knockoffs, when combined with thresholded-Lasso, can significantly improve the power (true positive rate) for a given false discovery rate (FDR) compared to using Lasso alone, even in its "oracle" (idealized) form.
  • AMP Theory is Powerful: The paper demonstrates the remarkable power of AMP theory to provide exact asymptotic predictions for complex statistical procedures like knockoff-based variable selection.
  • Cross-Validation is Near-Optimal: For i.i.d. covariates, tuning the regularization parameter ($\lambda$) for knockoff procedures using cross-validation on the augmented design matrix is shown to be nearly optimal, providing a practical way to achieve good performance.
  • Type S Error Analysis is Possible: The framework can be extended to analyze Type S errors, offering insights into the accuracy of sign determination for selected variables, which is crucial in many applications.
  • Thresholded-Lasso Outperforms Lasso: The analysis consistently shows that thresholded-Lasso, especially with knockoffs, offers a better trade-off between FDR and TPP than standard Lasso.

Key Results

  • The paper provides exact asymptotic predictions for the FDP and TPP of Model-X knockoff versions of Lasso and thresholded-Lasso selection procedures.
  • It demonstrates that thresholded-Lasso with Model-X knockoffs can achieve significantly better power-FDR trade-offs than Lasso, even breaking through theoretical lower bounds for Lasso.
  • A cross-validation procedure for selecting the tuning parameter $\lambda$ for LCD-knockoffs is proposed and shown to be nearly optimal in the i.i.d. setting.
  • The framework is extended to analyze Type S errors, showing that knockoff procedures can control the false sign proportion (FSP) and provide a notion of power for sign classification.

Limitations and Future Work

  • i.i.d. Covariate Assumption: While Model-X knockoffs are generally applicable, the power analysis in this paper is currently restricted to the i.i.d. Gaussian covariate setting. The authors express hope that this analysis can be extended to more general settings in future work.
  • Computational Complexity: While AMP theory provides theoretical insights, the practical implementation and computational cost of these methods, especially for very large datasets, might be a consideration.
  • Generalization to Other Statistics: The paper focuses on ℓp-regularized estimators (Lasso and thresholded-Lasso). Extending the analysis to other types of statistics or more complex models could be a future direction.
  • Multiple Knockoffs: The paper mentions the possibility of extending the theory to multiple knockoffs for each original variable, which could offer further improvements.

Summary updated

Abstract

Variable selection properties of procedures utilizing penalized-likelihood estimates is a central topic in the study of high-dimensional linear regression problems. Existing literature emphasizes the quality of ranking of the variables by such procedures as reflected in the receiver operating characteristic curve or in prediction performance. Specifically, recent works have harnessed modern theory of approximate message-passing (AMP) to obtain, in a particular setting, exact asymptotic predictions of the type I, type II error tradeoff for selection procedures that rely on ℓp-regularized estimators. In practice, effective ranking by itself is often not sufficient because some calibration for Type I error is required. In this work, we study theoretically the power of selection procedures that similarly rank the features by the size of an ℓp-regularized estimator, but further use Model-X knockoffs to control the false discovery rate in the realistic situation where no prior information about the signal is available. In analyzing the power of the resulting procedure, we extend existing results in AMP theory to handle the pairing between original variables and their knockoffs. This is used to derive exact asymptotic predictions for power. We apply the general results to compare the power of the knockoffs versions of Lasso and thresholded-Lasso selection, and demonstrate that in the i.i.d. covariate setting under consideration, tuning by cross-validation on the augmented design matrix is nearly optimal. We further demonstrate how the techniques allow to analyze also the Type S error, and a corresponding notion of power, when selections are supplemented with a decision on the sign of the coefficient.
BibTeX
@article{weinstein2023power,
  author = {Asaf Weinstein and Weijie Su and Małgorzata Bogdan and Rina Foygel Barber and Emmanuel J. Candès},
  title = {{A power analysis for model-X knockoffs with ℓp-regularized statistics}},
  journal = {The Annals of Statistics},
  year = {2023},
  doi = {10.1214/23-aos2274},
}
Figure