Introduction
Random Forests (RFs) are widely regarded as a robust tool for high-dimensional data analysis. However, they are not immune to the "curse of dimensionality." When datasets contain thousands of features but only a small subset is truly informative, RFs can become computationally inefficient and prone to decreased predictive accuracy. Traditional feature selection methods, such as the permutation-based Boruta algorithm or minimum-depth-based approaches, often lack rigorous theoretical foundations, exhibit bias toward features with many distinct values, or fail to scale to massive datasets.
This paper introduces NFSRD (Nonparametric Feature Selection by Random Forests and Deep Neural Networks), a hybrid framework designed to address these challenges. By combining the ranking capabilities of Random Forests with the statistical rigor of deep-learning-based hypothesis testing, NFSRD provides a computationally efficient and theoretically grounded method for identifying relevant features in complex datasets.
The NFSRD Framework
The NFSRD algorithm operates through a two-stage process: ranking and sequential selection.
1. Bias-Corrected Feature Importance (BCFI)
The first stage focuses on ranking features while correcting for common biases. Standard RF importance measures often favor categorical variables with many levels. NFSRD mitigates this by using "shadow features"—randomly shuffled versions of the original features—to establish a baseline for importance.
To ensure scalability, the authors implement a subsampling strategy. Instead of processing the entire dataset, importance is calculated on smaller subsets (e.g., 400 to 1,000 instances). This significantly reduces memory overhead and processing time without sacrificing the quality of the initial ranking.
2. Forward Selection via MMD-D
Once features are ranked, the algorithm employs a forward selection process. The critical innovation here is the stopping criterion, which utilizes Maximum Mean Discrepancy with Deep Kernels (MMD-D).
Rather than testing features in isolation, NFSRD performs a sequential hypothesis test on model residuals. It compares the residuals of a model using the top $K$ features against a model using the full feature set. If a Deep Neural Network (acting as a learned kernel) cannot distinguish between the two sets of residuals, the algorithm concludes that the remaining features do not contribute unique information and terminates the selection. This deep-kernel approach allows the test to remain sensitive to subtle, non-linear dependencies that traditional Gaussian kernels might miss in high-dimensional spaces.
Experimental Results
The authors validated NFSRD using both synthetic simulations and real-world data, comparing it against established methods like Boruta and the Minimum Depth method.
Synthetic Benchmarks
In simulations involving 400 features across various mathematical models (linear, non-additive, and trigonometric), NFSRD demonstrated superior precision. In a standard linear model setup:
- Boruta included an average of 2.17 useless features.
- Minimum Depth included an average of 34.88 useless features.
- NFSRD included only 0.3 to 0.4 useless features.
Furthermore, NFSRD proved significantly faster. For the largest synthetic tests, NFSRD completed the selection in under 50 seconds, while Boruta required over 125 seconds.
Real-World Application: Superconductivity
The method was applied to a superconductivity dataset (21,263 instances, 81 features) to predict critical temperatures. While the Boruta algorithm failed to provide any dimensionality reduction—selecting all 81 features as "useful"—NFSRD identified a concise set of 8 to 11 features. These features aligned closely with the top predictors identified by XGBoost, demonstrating that NFSRD can extract a highly informative signal from noisy, real-world data.
Discussion and Limitations
While NFSRD offers a powerful alternative to existing methods, it is subject to certain constraints. The algorithm's performance is sensitive to the subsample size ($m_0$); if the sample is too small relative to the signal-to-noise ratio, it may fail to detect relevant features. Additionally, as a forward-selection (greedy) method, it may struggle with extremely complex interactions where features only provide value when selected as a large, collective group.
Despite these limitations, NFSRD represents a significant step forward in nonparametric feature selection. Its ability to leverage deep learning for statistical testing on residuals provides a robust framework that can potentially be extended to other machine learning architectures beyond Random Forests.
Summary updated