The Holdout Randomization Test: Principled and Easy Black Box Feature Selection
Lists
Only you see this
Abstract
We consider the problem of feature selection using black box predictive models. For example, high-throughput devices in science are routinely used to gather thousands of features for each sample in an experiment. The scientist must then sift through the many candidate features to find explanatory signals in the data, such as which genes are associated with sensitivity to a prospective therapy. Often, predictive models are used for this task: the model is fit, error on held out data is measured, and strong performing models are assumed to have discovered some fundamental properties of the system. A model-specific heuristic is then used to inspect the model parameters and rank important features, with top features reported as discoveries. However, such heuristics provide no statistical guarantees and can produce unreliable results. We propose the holdout randomization test (HRT) as a principled approach to feature selection using black box predictive models. The HRT is similar to a permutation test, where each random reshuffling is a draw from the complete conditional distribution of the feature being tested. The HRT is model agnostic and produces a valid $p$-value for each feature, enabling control over the false discovery rate (or Type I error) for any predictive model. Further, the HRT is computationally efficient and, in simulations, has greater power than a competing knockoffs-based approach. Code is available at this https URL.
BibTeX
@article{tansey2018holdout,
author = {Wesley Tansey and Victor Veitch and Haoran Zhang and Raúl Rabadán and David M. Blei},
title = {{The Holdout Randomization Test: Principled and Easy Black Box Feature Selection}},
journal = {Journal of Computational and Graphical Statistics},
year = {2018},
url = {https://openreview.net/forum?id=aIRiMfcgbE},
}| Significance Tests of Feature Relevance for a Black-Box Learner | arXiv · 2021 |
| Fast and Powerful Conditional Randomization Testing via Distillation | arXiv · 2020 |
| Black Box FDR | arXiv · 2018 |
| A general framework for inference on algorithm-agnostic variable importance | Journal of the American Statistical Association · 2021 |
| Statistically Valid Variable Importance Assessment through Conditional Permutations | arXiv · 2023 |
| Knockoffs for the Mass: New Feature Importance Statistics with False Discovery Guarantees. | arXiv · 2018 |