TL;DR
This paper introduces "robust-equitability" as a new criterion for dependence measures, aiming to capture signal strength regardless of relationship type or noise level. They propose the Robust Copula Dependence (RCD) measure, which is shown to be both robustly equitable and easier to estimate than alternatives like mutual information, leading to more reliable feature selection.
What Problem Does This Paper Address?
In machine learning, not all input features are equally useful. Feature selection aims to identify the most relevant ones to improve model performance and reduce computational cost. The effectiveness of feature selection heavily relies on the "dependence measure" used to quantify the relationship between features and the target variable. Traditional measures like Pearson correlation only capture linear relationships, while more advanced measures like mutual information can capture non-linearities but are notoriously difficult to estimate reliably, especially with varying sample sizes or complex noise structures. This paper addresses the need for a dependence measure that is not only sensitive to non-linear relationships but also robust to different noise models and sample size variations, ensuring consistent and reliable feature ranking.
Methods
The authors introduce a new concept called robust-equitability. This builds upon existing notions of "equitability" (treating linear and non-linear relationships equally) and "self-equitability" (invariance to deterministic transformations in an additive noise model). Robust-equitability extends this to a "mixture noise model," where a deterministic signal is hidden within background noise, requiring the measure to reflect the signal's proportion.
They identify the Robust Copula Dependence (RCD) measure as satisfying this robust-equitability. RCD is based on the $L_1$-distance between the copula density and the uniform density (representing independence). They also theoretically prove that RCD is significantly easier to estimate than mutual information (MI) and a related measure, CD2, by analyzing their minimax convergence rates. Practical estimators for RCD using Kernel Density Estimation (KDE) and k-Nearest Neighbors (KNN) are proposed and analyzed.
Datasets
The paper uses synthetic datasets generated with various functional relationships (linear, square root, cubic, quadratic, circle, sine, etc.) and different noise models (additive and mixture noise) and sample sizes. They also evaluate their method on real-world datasets, including:
- StatLib stock data: For comparing feature relevance in predicting stock prices.
- KEGG metabolic reaction network data: For selecting features related to 'Characteristic path length'.
- Nine benchmark datasets from the UCI Machine Learning Repository: Used for evaluating mRMR-based feature selection performance.
Key Insights and Ideas
- Robust-Equitability is Crucial: The paper argues that for practical feature selection, especially with real-world data exhibiting mixed noise and varying sample sizes, a measure needs to be not just self-equitable but also robust-equitable. This ensures that the measure truly reflects the underlying signal strength, not just the functional form or sample size.
- MI and CD2's Estimation Difficulty: The theoretical analysis reveals that MI and CD2 have infinite minimax risk over certain classes of distributions, explaining why practitioners find them difficult to estimate reliably. This difficulty stems from their non-robust-equitability.
- RCD's Practical Advantage: RCD, being robust-equitable, is shown to be consistently estimable and provides stable rankings across different sample sizes and noise types, making it a more reliable choice for feature selection.
- Equitability and Noise Models: The paper highlights how different equitability definitions are tied to specific noise models (additive vs. mixture), and robust-equitability is the only one that bridges both.
Key Results
- Theoretical Proofs: The paper provides theoretical proofs demonstrating that RCD is robust-equitable, while MI and CD2 are not. It also proves the statistical difficulty of estimating MI and CD2 via minimax rates.
- Consistent Estimation: RCD is shown to be consistently estimable, with practical KDE and KNN estimators provided.
- Experimental Validation:
- Synthetic experiments show RCD provides stable rankings across different functional forms and noise types, unlike MI and CD2 which are sensitive to sample size and functional form.
- Real-world data experiments (stock data, metabolic network data) confirm RCD's ability to correctly rank features based on their true relevance, even with unequal sample sizes.
- mRMR-based feature selection using RCD consistently yields lower Mean Squared Error (MSE) on synthetic and benchmark datasets compared to other measures, indicating better feature selection performance. For instance, on a synthetic dataset, RCD correctly identified the top two features, leading to the lowest MSE. On nine real-world datasets, RCD was among the top-performing measures in 8 out of 9 cases.
Limitations and Future Work
- High-Dimensional Estimation: The paper acknowledges that RCD estimation in high-dimensional cases can still be challenging due to high-dimensional density estimation, similar to MI.
- Nonparametric RCD Estimation: Improving nonparametric RCD estimation remains an ongoing research effort, which could lead to wider applications.
- Tuning Parameter Selection: While practical estimators for RCD are provided, the selection of tuning parameters (bandwidth for KDE, number of neighbors for KNN) is based on empirical simulations, and further theoretical work could refine this.
Summary updated