TL;DR
This paper introduces KI-LAND, a novel method for automatically discovering and selecting both linear and nonlinear variables in partially linear models (PLMs). KI-LAND controls the false discovery rate (FDR) for both types of variables, offering theoretical guarantees and demonstrating strong empirical performance across various scenarios.
What Problem Does This Paper Address?
Partially linear models (PLMs) are powerful tools in statistics and machine learning, offering a balance between the interpretability of linear models and the flexibility of non-parametric models. They are widely used in fields like econometrics and biomedicine. A critical aspect of using PLMs is variable selection – identifying which covariates truly influence the outcome. While many methods exist for variable selection in PLMs, a significant gap has been the lack of robust control over the false discovery rate (FDR), meaning the rate at which irrelevant variables are incorrectly identified as significant. Furthermore, existing methods often struggle to simultaneously handle both linear and nonlinear variable selection in an automated fashion. This paper aims to fill this gap by proposing a new knockoff-based inference scheme that addresses FDR control for both linear and nonlinear components within PLMs, enabling automatic structure discovery.
Methods
The core of KI-LAND is an extension of the knockoff framework to partially linear models. The authors propose constructing "knockoff" variables that mimic the statistical properties of the original variables but are independent of the response. By comparing the importance scores of original variables to their knockoff counterparts, they can identify significant variables while controlling the FDR.
Key methodological contributions include:
- PLM Decomposition: The paper leverages the decomposition of functions in Reproducing Kernel Hilbert Spaces (RKHS) to separate each component function into a linear part and a nonlinear part. This allows for distinct treatment and selection of linear and nonlinear variables.
- KI-LAND Procedure: This involves a subsampling strategy to estimate importance scores for linear and nonlinear components. The procedure iteratively estimates model weights and solves optimization problems to identify potential linear and nonlinear variables.
- Knockoff Statistics: Based on the estimated importance scores, knockoff statistics are computed to quantify the difference in importance between original variables and their knockoffs.
- Data-Dependent Thresholds: The method uses data-dependent thresholds to select variables, ensuring FDR control at a specified nominal level. Two thresholding strategies are presented, one classic and one more conservative.
- Knockoff Variable Construction: KI-LAND supports both model-X knockoffs (suitable for multivariate normal data) and deep knockoffs (for more general distributions), offering flexibility in data handling.
Datasets
The paper uses simulated data for evaluation. The data is generated based on a partially linear model with specified linear, nonlinear, and noise components. The authors systematically vary sample sizes, variable dimensions, and correlation levels to test the robustness and performance of KI-LAND under different scenarios. No new benchmark datasets are introduced.
Key Insights and Ideas
- Unified FDR Control: The ability to control FDR simultaneously for both linear and nonlinear variables in PLMs is a significant advancement, simplifying the variable selection process.
- Automatic Structure Discovery: KI-LAND automates the identification of which variables are linear and which are nonlinear, removing the need for prior knowledge about the functional form of each covariate's effect.
- Theoretical Guarantees: The paper provides theoretical assurances for both FDR controllability and statistical power, which are crucial for building confidence in the method's reliability.
- Flexibility in Knockoff Construction: The support for both model-X and deep knockoffs makes KI-LAND applicable to a wider range of data distributions.
Key Results
The experimental results demonstrate that KI-LAND effectively controls the false discovery rate (FDR) below the target level (q=0.2) across various settings of sample size, variable dimension, and correlation. The method also achieves good statistical power, which tends to increase with sample size and decrease with increasing variable dimension or correlation. KI-LAND is shown to be competitive with existing methods like KKO and Model-X, particularly in its ability to handle both linear and nonlinear variable selection simultaneously. While KI-LAND might have slightly weaker power than KKO in some scenarios, it offers a more comprehensive solution by distinguishing between linear and nonlinear effects.
Limitations and Future Work
The authors acknowledge that the theoretical power analysis is asymptotic, meaning it holds for large sample sizes. While the paper provides theoretical guarantees for FDR control even for finite sample sizes, the power analysis is more focused on the large-sample behavior. The paper also notes that KI-LAND has weaker power than KKO in some experimental settings, suggesting potential areas for improvement. Future work could explore extending KI-LAND to more complex model structures or investigating its performance in real-world applications with high-dimensional, complex datasets.
Summary updated