Principles of Noise Robust Feature Selection

72 concepts70 papers1 proposal

High-dimensional datasets are inherently messy. Whether dealing with single-cell RNA sequencing, clinical health records, or sensor logs from physical machinery, data is frequently contaminated by measurement errors, missing values, and irrelevant features.

As Kenji Kira et al. observed in The Feature Selection Problem: Traditional Methods and a New Algorithm, traditional feature selection algorithms are highly sensitive to noise. In high-dimensional spaces, random noise can easily create spurious correlations that look statistically significant. Standard statistical measures often fail to differentiate between these noise-induced artifacts and true structural relationships, as noted by Max Welling et al. in Products of Edge-perts. When a model indiscriminately fuses all available features, this uninformative data degrades predictive performance, a vulnerability highlighted by Kevin D. Tang et al. in Combining the Right Features for Complex Event Recognition.

To counteract this, noise-robust feature selection shifts the goal from simply finding features with high variance or correlation to explicitly isolating informative subspaces from background noise. Rather than relying on rigid, clean-data assumptions (Xingyu Wu et al. in Practical Markov Boundary Learning without Strong Assumptions), robust selection relies on several core architectural mechanisms.

Instance-Wise Selection and Reliability Pathways

Traditional feature selection is global: it selects a fixed subset of features to be used across an entire dataset. However, real-world noise is rarely uniform. A feature might be highly accurate for one sample but completely corrupted or missing for another.

To handle this heterogeneity, models can adaptively select features for each individual data point. Jinsung Yoon et al. demonstrate this intuition in INVASE: Instance-wise Variable Selection using Neural Networks. By using an actor-critic framework to discover feature subsets of varying sizes per instance, the model ensures that predictive accuracy remains stable even as the global volume of irrelevant information grows. Similarly, Chin Wang Cheong et al. apply instance-wise grouping in Explanation Beyond Individual Features: Instance-wise Feature Grouping for EHR Predictive Analytics to preserve explanation stability when electronic health records suffer from localized missingness and noise.

Global Selection:   [All Samples] ----> Uses Fixed Feature Subset (Vulnerable to sample-specific noise)

Instance-Wise:     Sample A --------> Evaluates Reliability Pathway ----> Uses Subset {X1, X3}
                    Sample B --------> Evaluates Reliability Pathway ----> Uses Subset {X2, X4}

A parallel approach decouples feature estimation from a reliability-based selection pathway. In Selective Integration: A Model for Disparity Estimation, Michael S. Gray et al. show that muting unreliable local experts prevents ambiguous or noisy visual data from corrupting downstream processing. This concept of channel suppression and adaptive feature re-extraction is also utilized by Rongqiang Guan et al. in High precision control moment gyroscope fault diagnosis via joint attention mechanism to maintain diagnostic accuracy in high-interference environments.

Generalized Entropy and Robust Similarity Measures

Many classic information-theoretic feature selection methods rely on Shannon entropy. However, Shannon entropy treats all regions of a data distribution uniformly, making it highly sensitive to heavy-tailed noise and outliers.

By shifting to generalized entropy measures, such as Rényi or Tsallis entropy, algorithms gain a mathematical "dial"—a sensitivity parameter—that alters how the metric weights different parts of the distribution. As Snehalika Lall et al. demonstrate with their framework in sc-REnF: An Entropy Guided Robust Feature Selection for Single-Cell RNA-seq Data, adjusting this parameter allows the model to focus on dense, informative regions while effectively ignoring the technical noise and extreme sparsity common in single-cell biological data.

$$\text{Shannon Entropy treats all points equally vs. Generalized Entropy downweights outliers via parameter } q$$

Similarly, replacing standard mean-squared error objectives with robust similarity metrics like correntropy insulates the selection process from non-Gaussian noise. Badong Chen et al. combine correntropy with automatic relevance determination in Correntropy-Based Logistic Regression with Automatic Relevance Determination for Robust Sparse Brain Activity Decoding. Because correntropy is localized—meaning it emphasizes samples where variables are close and ignores large, outlier-driven deviations—it prevents the model from mistakenly selecting irrelevant features when processing noisy brain signals.

Bypassing Fixed Graph Structures

A common paradigm in unsupervised feature selection is to pre-compute a similarity graph (such as a $k$-nearest neighbor graph) to preserve the underlying geometry of the data. However, if the raw input data contains heavy noise, the initial distance calculations will be incorrect. The resulting graph then encodes corrupted neighbor relationships, permanently misleading any subsequent feature selection steps.

To break this feedback loop, noise-robust algorithms bypass fixed local structures entirely. Dongyoon Han et al. introduce this strategy in Unsupervised Simultaneous Orthogonal basis Clustering Feature Selection. By adaptively learning feature importance simultaneously with an orthogonal basis rather than relying on a static, pre-computed graph, their method avoids being trapped by initial data artifacts, resulting in a much more resilient selection process in high-dimensional spaces.

Implicit Selection and Complexity-Based Splitting

In tabular data classification, tree-based ensemble models consistently outperform deep learning architectures when uninformative features are present. As Leo Grinsztajn et al. explain in Why do tree-based models still outperform deep learning on typical tabular data?, tree-based models perform implicit feature selection during the learning process. While neural networks can easily be distracted by noise columns during gradient descent, a tree-based split can simply choose to ignore uninformative columns entirely when determining optimal decision boundaries.

This concept is extended to density estimation by Lincen Yang in Conditional Density Estimation with Histogram Trees. By pairing density estimation with a complexity-based split criterion, the algorithm penalizes splits that do not yield a substantial informational gain. This prevents the model from splitting on noise, a frequent failure mode for traditional recursive partitioning algorithms in high-dimensional settings.

Robust Optimization and Regularization Cascades

When training data contains corrupted labels or heavy-tailed noise, standard Empirical Risk Minimization (ERM) fails because it optimizes for average performance. A small percentage of highly corrupted samples can completely skew the global average, forcing the model to select irrelevant features to minimize the error on outliers (Yingjie Wang et al. in Tilted Sparse Additive Models; Changying Guo et al. in Robust Variable Selection and Estimation Based on Kernel Modal Regression).

Robust feature selection frameworks modify the optimization objective to insulate the selection mechanism from these errors:

Ultimately, noise-robust feature selection works by replacing fragile global assumptions with adaptive, localized, and complexity-penalized evaluation mechanisms. Whether by evaluating feature reliability on a per-sample basis or by utilizing entropy measures that downweight outliers, these methods ensure that machine learning models extract true structural signals from high-dimensional noise.

Key Foundational Work

The Feature Selection Problem: Traditional Methods and a New Algorithm (1992) — contributed 1 concept elements to this direction.
Selective Integration: A Model for Disparity Estimation (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_noise_robust_feature_selection_a6d4d56d,
  title = {noise-robust feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-robust-feature-selection/rdr_a6d4d56de24b4ef485ca2500bea15401},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_noise_robust_feature_selection_a6d4d56d_v1,
  title = {noise-robust feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-robust-feature-selection/rdr_a6d4d56de24b4ef485ca2500bea15401/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake