Model-free Feature Screening with Projection Correlation and FDR Control with Knockoff Features

Wanjun Liu, Yuan Ke, Runze LiarXiv 2019 · arXiv

Abstract

This paper proposes a model-free and data-adaptive feature screening method for ultra-high dimensional datasets. The proposed method is based on the projection correlation which measures the dependence between two random vectors. This projection correlation based method does not require specifying a regression model and applies to the data in the presence of heavy-tailed errors and multivariate response. It enjoys both sure screening and rank consistency properties under weak assumptions. Further, a two-step approach is proposed to control the false discovery rate (FDR) in feature screening with the help of knockoff features. It can be shown that the proposed two-step approach enjoys both sure screening and FDR control if the pre-specified FDR level $\alpha$ is greater or equal to $1/s$, where $s$ is the number of active features. The superior empirical performance of the proposed methods is justified by various numerical experiments and real data applications.

Introduction

In the era of "Big Data," researchers frequently encounter datasets where the number of features ($p$) vastly exceeds the number of observations ($n$). Identifying the relevant variables within these ultra-high dimensional spaces—the "needle in the haystack"—is a fundamental challenge in modern statistics. Traditional feature screening methods, such as Sure Independence Screening (SIS), have historically relied on two restrictive assumptions: the existence of a linear relationship between features and the response, and specific distributional properties (often Gaussian) of the data.

When these assumptions are violated—such as in the presence of nonlinear dependencies or heavy-tailed distributions with frequent outliers—traditional methods often fail. Furthermore, even when features are ranked correctly, researchers face the "threshold problem": determining exactly how many features to retain without inflating the False Discovery Rate (FDR). This paper introduces PC-Screen and PC-Knockoff, a robust framework designed to address these limitations through model-free screening and rigorous error control.

Methodology

PC-Screen: The Model-Free Filter

The core of the proposed screening method is Projection Correlation (PC). Unlike standard correlation coefficients that measure linear associations, PC measures the dependence between two random vectors regardless of their distribution or the complexity of their relationship.

PC-Screen offers several distinct advantages:

  1. Model-Agnosticism: It does not require the user to specify a regression model (e.g., linear, logistic, or Poisson). It is equally effective whether the underlying relationship is a straight line or a complex sine wave.
  2. Robustness: It is naturally resistant to heavy-tailed errors (such as those following a Cauchy distribution) that typically derail mean-based screening methods.
  3. Multivariate Support: It handles multivariate responses—predicting multiple dependent variables simultaneously—without requiring significant modifications.

PC-Knockoff: Achieving FDR Control

To solve the threshold problem, the authors propose a two-step procedure utilizing Knockoff Features. The process begins by splitting the data. A portion of the data is used to narrow the thousands of potential features down to a manageable, albeit "over-fitted," subset.

For every real feature in this subset, the algorithm constructs a "knockoff"—a synthetic variable that mimics the correlation structure of the original feature but is known to be independent of the response. By comparing the performance of real features against their synthetic counterparts, the algorithm can statistically determine a cutoff point. This ensures the FDR remains below a pre-specified level ($\alpha$), provided that $\alpha \geq 1/s$, where $s$ is the number of true active features.

Empirical Results

Simulation Studies

The authors validated the method across various scenarios, including linear, Poisson, and highly nonlinear models. A key finding was the method's performance in the presence of heavy-tailed noise. While competitors like SIS broke down when faced with extreme outliers, PC-Screen remained stable, consistently ranking true active features at the top of the list.

In nonlinear simulations, PC-Screen successfully recovered the true active features with a model size nearly identical to the ground truth (approximately 4). In contrast, competing methods often required retaining hundreds or thousands of features to capture the same signal, significantly increasing the noise in the final model.

Real-World Application: Supermarket Sales

The method was applied to a supermarket dataset ($n=464$, $p=6398$) to predict daily customer traffic based on the sales volumes of thousands of products. The PC-Knockoff procedure selected 12 key products. This parsimonious model achieved an $R^2$ of 0.86, outperforming previous state-of-the-art methods ($R^2 = 0.85$) that utilized more complex or larger feature sets.

Discussion and Limitations

While PC-Screen provides a powerful tool for high-dimensional analysis, it is not without trade-offs. The calculation of Projection Correlation has a computational complexity of $O(n^3)$. While this is manageable for "large $p$, small $n$" problems, it may become a bottleneck as the number of observations ($n$) increases into the tens of thousands.

Additionally, the current reliance on data splitting, while mathematically elegant for ensuring FDR control, effectively reduces the sample size available for each step. Future work may explore "data recycling" techniques to improve the power of the test. Finally, the quality of the FDR control is linked to the quality of the second-order knockoffs; extremely complex, non-Gaussian dependencies between features may require more sophisticated knockoff generation techniques in the future.

Conclusion

PC-Screen and PC-Knockoff represent a significant step forward in high-dimensional feature selection. By moving away from rigid model assumptions and providing a reliable mechanism for FDR control, this framework allows researchers to extract meaningful insights from complex, noisy, and ultra-high dimensional data with greater confidence.

Summary updated

Abstract

This paper proposes a model-free and data-adaptive feature screening method for ultra-high dimensional datasets. The proposed method is based on the projection correlation which measures the dependence between two random vectors. This projection correlation based method does not require specifying a regression model and applies to the data in the presence of heavy-tailed errors and multivariate response. It enjoys both sure screening and rank consistency properties under weak assumptions. Further, a two-step approach is proposed to control the false discovery rate (FDR) in feature screening with the help of knockoff features. It can be shown that the proposed two-step approach enjoys both sure screening and FDR control if the pre-specified FDR level $\alpha$ is greater or equal to $1/s$, where $s$ is the number of active features. The superior empirical performance of the proposed methods is justified by various numerical experiments and real data applications.
BibTeX
@misc{liu2019model,
  author = {Wanjun Liu and Yuan Ke and Runze Li},
  title = {{Model-free Feature Screening with Projection Correlation and FDR Control with Knockoff Features}},
  year = {2019},
  eprint = {1908.06597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.06597},
}
Figure