Abstract
Feature selection aims to select relevant features to improve the performance of predictors. Many feature selection methods depend on the choice of dependence measures. To select features that have complex nonlinear relationships with the target variable, this paper introduces Robust Copula Dependence (RCD), a robust-equitable measure that remains stable across varying noise levels and sample sizes.
Introduction
In machine learning, feature selection is critical for improving model performance by identifying the most informative variables. This process relies heavily on "dependence measures" to rank features. However, traditional measures often fail in two key areas: equitability and robustness. Simple measures like Pearson correlation are limited to linear relationships, while more complex ones like Mutual Information (MI) are theoretically equitable but difficult to estimate accurately from finite data.
Furthermore, many measures are highly sensitive to noise and scale. A significant challenge is "mixture noise," where signal is buried in background noise, or scenarios where features have vastly different sample sizes. Traditional measures might incorrectly rank a weak feature with many samples higher than a strong feature with fewer samples.
Methods: Robust Copula Dependence (RCD)
The authors propose Robust Copula Dependence (RCD), leveraging the mathematical framework of copulas. Copulas allow for the separation of the dependency structure between variables from their individual marginal distributions. RCD is defined as half the $L_1$ distance between the copula density and the uniform (independence) density, making it mathematically equivalent to Silvey’s Delta.
To make RCD practical, the authors developed a KNN-based estimator. Unlike Mutual Information or kernel-based methods like HSIC, which can become unstable or even infinite under certain noise conditions, RCD is bounded between 0 and 1. This boundedness ensures stability and allows for fair comparisons across features with different scales and sample sizes.
Key Results and Insights
The effectiveness of RCD was evaluated using both synthetic and real-world datasets, including stock prices, metabolic networks, and housing data.
- Stability and Consistency: The authors demonstrate that theoretical values for MI and HSIC can become infinite in the presence of certain noise types, leading to unstable estimates. RCD avoids this "infinite problem," remaining consistent as sample sizes increase from 300 to 10,000.
- Ranking Accuracy: In synthetic tests, RCD was the only measure to correctly rank features by their true predictive power across varying sample sizes.
- Real-World Performance: In benchmarks using the mRMR (minimum Redundancy Maximum Relevance) framework, RCD-based selection achieved the lowest mean squared error (MSE) in five out of six datasets, outperforming MI and HSIC.
Conclusion
RCD provides a robust and equitable alternative for feature selection in complex, noisy datasets. While the current KNN-based estimator has a computational complexity of $O(n^2 \log n)$, it offers significant reliability for big data applications where traditional measures falter. Future research will focus on optimizing this estimator for larger scales and extending the measure to capture multivariate interactions.
Summary updated