TL;DR
This paper introduces the Holdout Randomization Test (HRT), a novel method for feature selection in "black box" predictive models. The HRT provides a statistically sound way to identify important features by generating valid p-values, overcoming limitations of heuristic approaches and offering a computationally feasible alternative to existing randomization tests.
What Problem Does This Paper Address?
In many scientific and data analysis tasks, the goal is to identify which features (variables) are most important for predicting an outcome. A common approach is to build a complex predictive model (a "black box" model, like a neural network or random forest) and then use heuristic methods (e.g., feature importance scores from the model) to rank features. However, these heuristics often lack statistical guarantees and can lead to false discoveries, especially when features are correlated. Existing principled methods for feature selection, like Conditional Randomization Tests (CRTs), are often computationally too expensive for large datasets. This paper addresses the need for a statistically rigorous, yet computationally practical, method for feature selection in the context of black box models.
Methods
The core contribution is the Holdout Randomization Test (HRT). It's a specialized version of the Conditional Randomization Test (CRT) that uses data splitting (a train-test split) to make computation feasible. Here's how it works:
- Train a Predictive Model: A predictive model (e.g., random forest, lasso) is trained on a portion of the data.
- Evaluate on Held-Out Data: The model's performance (measured by an "empirical risk" function, like prediction error) is evaluated on a separate test set.
- Generate Null Samples: For each feature being tested, the HRT generates "null" versions of that feature. These null features are designed to be conditionally independent of the outcome, given all other features. This is done by sampling from the complete conditional distribution of the feature.
- Compare Performance: The predictive model's performance is re-evaluated on the test set, but with the original feature replaced by its null version.
- Calculate p-value: The p-value for a feature is the proportion of null samples that result in a worse model performance than the original data. A small p-value suggests the feature is truly predictive.
The authors also propose several extensions to the HRT:
- Cross-Validation HRT (CV-HRT): Uses M-fold cross-validation to leverage the entire dataset for training and testing, increasing power.
- Approximate CV-HRT: A faster version of CV-HRT that merges test statistics without correction, often yielding higher power.
- Holdout Permutation Test (HPT): Adapts permutation testing to the HRT framework for better scalability.
- Holdout Grid Test (HGT): Speeds up computation by creating a finite grid approximation of the conditional distribution and caching risk scores.
Datasets
The paper uses synthetic datasets for simulations and two real-world datasets for case studies:
- Drug response experiment: Genomic data from cancer cell lines and their response to an anti-cancer drug (from Barretina et al., 2012).
- Olfactory perception experiment: Molecular structure data and human ratings of perceived fragrance (from Keller et al., 2017).
The authors also mention using a benchmark from Liang et al. (2018) for evaluating performance when conditional distributions are unknown.
Key Insights and Ideas
- Principled Feature Selection for Black Boxes: The HRT provides a statistically sound way to select features from complex, non-linear models where traditional methods fail or are too slow.
- Data Splitting is Key: The HRT's use of a holdout set is crucial for making computationally intensive randomization tests practical.
- Heuristics Can Be Misleading: The case studies highlight how common heuristic feature importance measures can disagree with statistically validated findings, potentially leading to incorrect scientific conclusions.
- Extensions Boost Power and Speed: The proposed extensions (CV-HRT, HGT) significantly improve the HRT's power and computational efficiency, making it more applicable to real-world problems.
- Predictive Model Quality Matters: The HRT's power is directly related to how well the chosen predictive model captures the underlying data relationships. Better models lead to better feature selection.
Key Results
- Improved Power: The HRT and its extensions generally show higher power than model-X knockoffs, especially when using the same predictive model (e.g., lasso).
- Computational Feasibility: The HRT extensions (HGT, CV-HRT) significantly reduce computation time, with HGT being orders of magnitude faster than the basic HRT.
- Valid p-values: The HRT and its valid extensions (like CV-HRT) produce statistically valid p-values, controlling the false discovery rate (FDR) at the desired level.
- Case Study Findings: In real-world examples, the HRT identified features that were missed by heuristic methods, suggesting new potential targets for therapy or understanding.
- Model Choice Impact: The quality of the predictive model directly impacts feature selection power, with models that generalize better leading to higher power.
Limitations and Future Work
- Estimating Complete Conditionals: The HRT relies on estimating complete conditional distributions, which can be challenging. The paper proposes a heuristic calibration method but acknowledges that more robust theoretical work is needed.
- Optimal Data Splitting: Future work could explore optimal strategies for allocating data to training and testing sets to maximize power.
- Arbitrarily Fine-Grained Questions: The HRT's framework could be extended to answer more complex questions, such as feature dependencies within specific subsets of the data.
- Computational Cost of Inference: While extensions improve speed, applying the HRT to extremely large feature sets or complex models can still be computationally demanding.
- Model-Specific Power: The paper notes that the power discrepancy between lasso coefficient knockoffs and empirical risk might be specific to their simulations, and further work is needed to establish the relative power of different statistics.
Summary updated