Redundancy Aware Feature Selection

270 concepts230 papers2 proposals

Imagine you are building a team of experts to solve a complex puzzle. If you select your team by testing each candidate individually and picking the top five scorers, you might end up with five people who went to the same school, read the same books, and possess identical knowledge. They will excel at the same parts of the puzzle but leave massive blind spots elsewhere.

This is the fundamental flaw of traditional feature selection methods. As Yang et al. point out, standard metrics often prioritize individual variable relevance without accounting for information overlap between features. When features are ranked and selected purely by their individual correlation with the target variable, the resulting subset is frequently bloated with "copycat" features (Poignard et al.). These redundant variables increase computational cost, memory usage, and the risk of overfitting without contributing any novel predictive power (Gong et al.; Anand et al.).

This issue spans across domains. In deep neural networks, highly predictive features are often deeply redundant, offering little incremental gain when combined (Kuemmerer et al.). In computer vision, selecting the highest-performing individual filters results in a pool that captures the exact same visual patterns, severely limiting the detector's efficiency (Ahmed et al.). To build truly efficient models, we must shift our focus from how much information a feature carries to what unique information it provides.

Quantifying Information Overlap

To eliminate redundancy, we first need a way to measure it. A common mathematical foundation for this is information theory, specifically Joint Mutual Information (JMI) and conditional mutual information. Instead of asking "Does feature $X$ predict target $Y$?", we ask "Does feature $X$ predict target $Y$ given that we already know feature $Z$?"

However, calculating these high-dimensional dependencies is computationally expensive. Researchers have developed several clever workarounds to isolate the "marginal contribution" of a feature:

  • Residual Analysis: In iterative selection, instead of evaluating raw features against the target, we can evaluate them against the residual of the current model. As demonstrated in column subset selection, calculating a residual matrix ensures that subsequent selection rounds focus entirely on what has not yet been captured (Singh et al.).
  • Geometric Manifolds: Rather than relying on simple linear correlations, methods like ManiFeSt treat features as points on a geometric manifold (Cohen et al.). By using Riemannian geometry to compare class-specific structures, the framework identifies complex multivariate interactions and filters out overlapping signals even when data is scarce.
  • Subspace Projection: In model interpretability and parameter-efficient fine-tuning, projecting latent representations onto orthogonal subspaces allows models to isolate distinct factors. For instance, Zhang et al. use deconvolutional layers to ensure that learned features in compressed subspaces remain distinct, maximizing the information captured within a tight parameter budget.

The Relevance-Redundancy Tradeoff

The goal of redundancy-aware feature selection is often framed as an optimization problem: maximize relevance to the target while minimizing redundancy among the selected features (the minimum Redundancy Maximum Relevance, or mRMR, framework).

But why not just eliminate all redundancy? Interestingly, Wu et al. introduce a crucial nuance: some redundancy is actually vital for a system's robustness. They argue that distinguishing between irrelevant noise and beneficial overlapping information allows for a more stable feature fusion process. The challenge is therefore not a blind elimination, but a careful balancing act.

To achieve this balance algorithmically, researchers often turn to submodularity and spectral regularization. Das et al. leverage matrix spectra to penalize highly correlated variables mathematically, ensuring that the selected subset captures the full breadth of the data. Similarly, Quon utilizes Determinantal Point Processes (DPPs) to explicitly encourage diversity among predictors, which significantly enhances model interpretability in domains like genomics where identifying distinct underlying biological mechanisms is paramount.

Beyond Flat Feature Sets

Modern applications show that redundancy doesn't just happen in a flat list of numbers; it occurs across complex structures:

  • Group-Level Redundancy: Features often carry categorical or structural relationships (e.g., genes belonging to the same biological pathway). Standard metrics fail here because they might select multiple features from the same functional category. Frameworks like GroupMRMR incorporate external group knowledge to ensure semantic diversity across different functional categories (Chan et al.).
  • Implicit vs. Explicit Redundancy: When enriching models with new features—such as adding semantic relationship graphs to depth estimation models (Xu et al.) or variable roles to code models (Hussain et al.)—much of the information may already be implicitly captured by the raw data or the network architecture. Without explicit orthogonalization, these expensive feature engineering efforts fail to provide any measurable benefit.

Ultimately, whether managing latency constraints in search engines (Lin et al.) or optimizing human effort in crowdsourced feature discovery (Takahama et al.), accounting for information overlap transforms feature selection from a simple ranking exercise into a strategic assembly of complementary signals.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 224 papers
  2. Version 1 · 2026-10-04 · 135 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_feature_redundancy_information_overlap_e2e36248,
  title = {feature redundancy (information overlap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-information-overlap/rdr_e2e3624837134e9b937f52c0125818df},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_feature_redundancy_information_overlap_e2e36248_v2,
  title = {feature redundancy (information overlap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-information-overlap/rdr_e2e3624837134e9b937f52c0125818df/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake