High-dimensional datasets are inherently messy. Whether dealing with single-cell RNA sequencing, clinical health records, or sensor logs from physical machinery, data is frequently contaminated by measurement errors, missing values, and irrelevant features.
As Kenji Kira et al. observed in The Feature Selection Problem: Traditional Methods and a New Algorithm, traditional feature selection algorithms are highly sensitive to noise. In high-dimensional spaces, random noise can easily create spurious correlations that look statistically significant. Standard statistical measures often fail to differentiate between these noise-induced artifacts and true structural relationships, as noted by Max Welling et al. in Products of Edge-perts. When a model indiscriminately fuses all available features, this uninformative data degrades predictive performance, a vulnerability highlighted by Kevin D. Tang et al. in Combining the Right Features for Complex Event Recognition.
To counteract this, noise-robust feature selection shifts the goal from simply finding features with high variance or correlation to explicitly isolating informative subspaces from background noise. Rather than relying on rigid, clean-data assumptions (Xingyu Wu et al. in Practical Markov Boundary Learning without Strong Assumptions), robust selection relies on several core architectural mechanisms.
Instance-Wise Selection and Reliability Pathways
Traditional feature selection is global: it selects a fixed subset of features to be used across an entire dataset. However, real-world noise is rarely uniform. A feature might be highly accurate for one sample but completely corrupted or missing for another.
To handle this heterogeneity, models can adaptively select features for each individual data point. Jinsung Yoon et al. demonstrate this intuition in INVASE: Instance-wise Variable Selection using Neural Networks. By using an actor-critic framework to discover feature subsets of varying sizes per instance, the model ensures that predictive accuracy remains stable even as the global volume of irrelevant information grows. Similarly, Chin Wang Cheong et al. apply instance-wise grouping in Explanation Beyond Individual Features: Instance-wise Feature Grouping for EHR Predictive Analytics to preserve explanation stability when electronic health records suffer from localized missingness and noise.
Global Selection: [All Samples] ----> Uses Fixed Feature Subset (Vulnerable to sample-specific noise)
Instance-Wise: Sample A --------> Evaluates Reliability Pathway ----> Uses Subset {X1, X3}
Sample B --------> Evaluates Reliability Pathway ----> Uses Subset {X2, X4}
A parallel approach decouples feature estimation from a reliability-based selection pathway. In Selective Integration: A Model for Disparity Estimation, Michael S. Gray et al. show that muting unreliable local experts prevents ambiguous or noisy visual data from corrupting downstream processing. This concept of channel suppression and adaptive feature re-extraction is also utilized by Rongqiang Guan et al. in High precision control moment gyroscope fault diagnosis via joint attention mechanism to maintain diagnostic accuracy in high-interference environments.
Generalized Entropy and Robust Similarity Measures
Many classic information-theoretic feature selection methods rely on Shannon entropy. However, Shannon entropy treats all regions of a data distribution uniformly, making it highly sensitive to heavy-tailed noise and outliers.
By shifting to generalized entropy measures, such as Rényi or Tsallis entropy, algorithms gain a mathematical "dial"—a sensitivity parameter—that alters how the metric weights different parts of the distribution. As Snehalika Lall et al. demonstrate with their framework in sc-REnF: An Entropy Guided Robust Feature Selection for Single-Cell RNA-seq Data, adjusting this parameter allows the model to focus on dense, informative regions while effectively ignoring the technical noise and extreme sparsity common in single-cell biological data.
$$\text{Shannon Entropy treats all points equally vs. Generalized Entropy downweights outliers via parameter } q$$
Similarly, replacing standard mean-squared error objectives with robust similarity metrics like correntropy insulates the selection process from non-Gaussian noise. Badong Chen et al. combine correntropy with automatic relevance determination in Correntropy-Based Logistic Regression with Automatic Relevance Determination for Robust Sparse Brain Activity Decoding. Because correntropy is localized—meaning it emphasizes samples where variables are close and ignores large, outlier-driven deviations—it prevents the model from mistakenly selecting irrelevant features when processing noisy brain signals.
Bypassing Fixed Graph Structures
A common paradigm in unsupervised feature selection is to pre-compute a similarity graph (such as a $k$-nearest neighbor graph) to preserve the underlying geometry of the data. However, if the raw input data contains heavy noise, the initial distance calculations will be incorrect. The resulting graph then encodes corrupted neighbor relationships, permanently misleading any subsequent feature selection steps.
To break this feedback loop, noise-robust algorithms bypass fixed local structures entirely. Dongyoon Han et al. introduce this strategy in Unsupervised Simultaneous Orthogonal basis Clustering Feature Selection. By adaptively learning feature importance simultaneously with an orthogonal basis rather than relying on a static, pre-computed graph, their method avoids being trapped by initial data artifacts, resulting in a much more resilient selection process in high-dimensional spaces.
Implicit Selection and Complexity-Based Splitting
In tabular data classification, tree-based ensemble models consistently outperform deep learning architectures when uninformative features are present. As Leo Grinsztajn et al. explain in Why do tree-based models still outperform deep learning on typical tabular data?, tree-based models perform implicit feature selection during the learning process. While neural networks can easily be distracted by noise columns during gradient descent, a tree-based split can simply choose to ignore uninformative columns entirely when determining optimal decision boundaries.
This concept is extended to density estimation by Lincen Yang in Conditional Density Estimation with Histogram Trees. By pairing density estimation with a complexity-based split criterion, the algorithm penalizes splits that do not yield a substantial informational gain. This prevents the model from splitting on noise, a frequent failure mode for traditional recursive partitioning algorithms in high-dimensional settings.
Robust Optimization and Regularization Cascades
When training data contains corrupted labels or heavy-tailed noise, standard Empirical Risk Minimization (ERM) fails because it optimizes for average performance. A small percentage of highly corrupted samples can completely skew the global average, forcing the model to select irrelevant features to minimize the error on outliers (Yingjie Wang et al. in Tilted Sparse Additive Models; Changying Guo et al. in Robust Variable Selection and Estimation Based on Kernel Modal Regression).
Robust feature selection frameworks modify the optimization objective to insulate the selection mechanism from these errors:
- Label-Flipping Models: Ata Kaban shows in Classification of mislabelled microarrays using robust sparse logistic regression that integrating a label-flipping probability directly into a sparse Bayesian classifier allows the model to maintain stable feature selection even when up to 30% of the training labels are incorrect.
- Reweighted Optimization: In Sparse Bayesian classification and feature selection for biological expression data with high correlations, Wei Pan demonstrates that reweighted optimization algorithms maintain high sensitivity in noisy environments, significantly reducing the false-negative rate when identifying highly correlated biomarkers.
- Regularization Cascades and Forward Selection: Iterative refinement mechanisms, such as the cascade proposed by Alon Zweig et al. in Hierarchical Regularization Cascade for Joint Learning, gradually adjust the incentive to share features across tasks. This ensures that performance degrades gracefully as input noise increases. Similarly, iterative forward selection processes, like the random field method introduced by Yalu Wen et al. in Risk Prediction Modeling of Sequencing Data Using a Forward Random Field Method, successfully exclude non-causal features by evaluating incremental predictive power rather than static global variance.
Ultimately, noise-robust feature selection works by replacing fragile global assumptions with adaptive, localized, and complexity-penalized evaluation mechanisms. Whether by evaluating feature reliability on a per-sample basis or by utilizing entropy measures that downweight outliers, these methods ensure that machine learning models extract true structural signals from high-dimensional noise.
Key Foundational Work
The Feature Selection Problem: Traditional Methods and a New Algorithm (1992) — contributed 1 concept elements to this direction.
Selective Integration: A Model for Disparity Estimation (1996) — contributed 1 concept elements to this direction.