Introduction
In machine learning, the "curse of dimensionality" presents a significant hurdle. When a model is trained on too many input features relative to the number of observations, it risks overfitting—capturing noise rather than the underlying signal. Input selection is the process of identifying the most relevant subset of features to mitigate this risk.
This study, originally presented at ESANN 2005, compares two model-independent techniques for feature selection: the Gamma Test (GT) and Mutual Information (MI). Unlike model-dependent "wrapper" methods that require training a model for every possible input combination, these criteria rank features based on mathematical properties of the data itself, offering significant computational efficiency.
Methodology
The authors evaluate two distinct mathematical philosophies for ranking input sets:
- The Gamma Test (GT): A non-linear technique that estimates the "noise floor" or the minimum Mean Square Error (MSE) achievable for a smooth model. By analyzing how outputs behave for "near-neighbor" inputs, GT identifies the input set that minimizes estimated noise.
- Mutual Information (MI): An information-theoretic metric measuring the statistical dependence between variables. Using a modern $k$-nearest neighbor estimator, the goal is to maximize the information shared between the selected inputs and the target output.
To validate these selections, a Least Squares Support Vector Machine (LS-SVM) is trained on the chosen features to measure actual prediction performance.
Experimental Results
Synthetic Noise Robustness
Using a 10-dimensional synthetic dataset where only four variables are relevant, the authors tested the robustness of both methods against increasing noise variance.
As shown in Figure 1, MI proved to be the more robust tool in extreme conditions, correctly identifying the relevant variables until noise variance reached 0.2940. In contrast, the Gamma Test began to fail earlier, losing accuracy once noise variance exceeded 0.1933.
Real-World Time Series Prediction
The methods were also applied to the Poland Electricity Dataset, a benchmark for predicting future electricity loads based on historical lags. Despite MI's superior robustness in synthetic tests, the Gamma Test proved to be the "sharper" tool for actual prediction accuracy.
The GT-selected regressor achieved a significantly lower Mean Square Error (0.00103) compared to the MI-selected regressor (0.00163). As illustrated in Figure 2, the predictions based on GT-selected inputs tracked the true electricity load values with high precision.
Conclusion
The study reveals a fundamental trade-off: while Mutual Information is more resilient to extreme noise, the Gamma Test consistently selects input sets that yield higher predictive accuracy in practical scenarios. GT's ability to provide a concrete estimate of the achievable MSE also offers a valuable target for model development that relative ranking metrics like MI cannot provide.
Summary updated