Kernel Knockoffs Selection for Nonparametric Additive Models

Xiaowu Dai, Xiang Lyu, Lexin LiJournal of the American Statistical Association 2022 · DOI

TL;DR

This paper introduces a novel method for selecting important variables in nonparametric additive models, a flexible class of models widely used in statistics and machine learning. Unlike previous methods, their approach guarantees control of the false discovery rate (FDR) even with limited sample sizes, a crucial feature for reliable inference.

What Problem Does This Paper Address?

Nonparametric additive models offer a powerful way to capture complex relationships between predictors and a response variable while maintaining interpretability. A key challenge in using these models is variable selection: identifying which predictor variables truly influence the response. While many methods exist, a significant limitation has been their inability to control the false discovery rate (FDR) – the expected proportion of selected variables that are actually irrelevant – unless the sample size is very large. This means that with smaller datasets, existing methods might falsely identify irrelevant variables as important. The "knockoff" framework has emerged as a promising solution for FDR control in linear models, but adapting it to the more complex nonparametric additive setting has been challenging.

Methods

The authors propose a novel "kernel knockoffs" selection procedure that integrates three key components:

  • Knockoffs: They construct "knockoff" variables that mimic the statistical properties of the original variables but are independent of the response. This is a core idea of the knockoff framework, allowing for FDR control.
  • Random Feature Mapping: To handle the nonparametric nature of the component functions in additive models, they use random feature mapping (inspired by Rahimi and Recht, 2007) to approximate these functions. This allows them to work with a finite-dimensional representation, making the problem more tractable.
  • Subsampling for Stability: To mitigate the additional randomness introduced by random feature mapping and improve selection stability, they employ a subsampling strategy. This involves repeatedly drawing subsamples of the data, performing the knockoff procedure, and aggregating the results to compute an importance score for each variable.

This combination allows them to define an importance score based on the difference in selection frequencies between original variables and their knockoffs across multiple subsamples. A knockoff filter is then applied to select variables while controlling the FDR.

Datasets

The paper does not introduce new datasets. It uses simulations to evaluate the proposed method and applies it to a publicly available brain imaging dataset (ADNI) to study Alzheimer's disease.

Key Insights and Ideas

  • Finite-Sample FDR Control for Nonparametric Models: This is the paper's central contribution. Achieving FDR control without requiring an infinite sample size is a significant step forward for reliable variable selection in flexible nonparametric settings.
  • Bridging the Gap: The method effectively bridges the gap between the powerful but FDR-uncontrolled methods for nonparametric models and the FDR-controlled but limited linear model knockoff methods.
  • Random Features as a Key Enabler: Random feature mapping is crucial for approximating the nonparametric functions and creating a structure that allows for the construction of valid knockoffs in this setting.
  • Subsampling for Robustness: The use of subsampling is essential for stabilizing the selection process in the face of the inherent randomness of the random feature mapping.

Key Results

  • Guaranteed FDR Control: The proposed kernel knockoffs method is theoretically guaranteed to control the FDR below any nominal level for any sample size.
  • Asymptotic Power: The method achieves a power that approaches one as the sample size tends to infinity.
  • Empirical Performance: Extensive simulations show that the proposed method (KKO) consistently controls the FDR at the nominal level while achieving competitive or superior power compared to existing methods (SPAM, LKO, RANK) across various settings (signal strength, predictor distributions, component functions, sample size, and dimension).
  • Robustness: The method demonstrates robustness to different predictor distributions and kernel choices.
  • Brain Imaging Application: The method successfully identified brain regions associated with Alzheimer's disease, aligning with existing literature.

Limitations and Future Work

  • Computational Cost: While the method is parallelizable, the computational cost can still be a factor, especially with a large number of subsampling replications.
  • Theoretical Investigation of Diverging Sparsity: The current theoretical guarantees assume a bounded number of nonzero component functions. The authors speculate that the method might work well even when the number of nonzero components grows with the sample size, but this requires further theoretical investigation.
  • Extension to More General Nonparametric Models: The authors briefly mention that their theoretical tools might be applicable to more general nonparametric models, such as those involving higher-order interactions, suggesting a potential avenue for future research.

Summary updated

Abstract

Thanks to its fine balance between model flexibility and interpretability, the nonparametric additive model has been widely used, and variable selection for this type of model has been frequently studied. However, none of the existing solutions can control the false discovery rate (FDR) unless the sample size tends to infinity. The knockoff framework is a recent proposal that can address this issue, but few knockoff solutions are directly applicable to nonparametric models. In this article, we propose a novel kernel knockoffs selection procedure for the nonparametric additive model. We integrate three key components: the knockoffs, the subsampling for stability, and the random feature mapping for nonparametric function approximation. We show that the proposed method is guaranteed to control the FDR for any sample size, and achieves a power that approaches one as the sample size tends to infinity. We demonstrate the efficacy of our method through intensive simulations and comparisons with the alternative solutions. our proposal thus makes useful contributions to the methodology of nonparametric variable selection, FDR-based inference, as well as knockoffs.
BibTeX
@article{dai2022kernel,
  author = {Xiaowu Dai and Xiang Lyu and Lexin Li},
  title = {{Kernel Knockoffs Selection for Nonparametric Additive Models}},
  journal = {Journal of the American Statistical Association},
  year = {2022},
  doi = {10.1080/01621459.2022.2039671},
}
Figure