Relevance-Redundancy Feature Optimization

19 concepts20 papers

When building machine learning models, more data does not always mean better performance. High-dimensional datasets—such as genomic profiles, satellite imagery, or dense medical scans—often contain hundreds or thousands of features. A naive approach to reducing this dimensionality is to evaluate each feature independently, rank them by how well they predict the target variable, and select the top performers.

However, this strategy frequently falls into the redundancy trap. If your top five features are highly correlated with each other, selecting all five adds virtually no new information to your model while increasing noise and computational complexity.

To solve this, the Minimum Redundancy Maximum Relevance (mRMR) framework optimizes a dual objective: it searches for a feature subset that maximizes predictive power regarding the target variable while simultaneously minimizing the statistical overlap among the selected features.

The Intuition: Assembling an Expert Team

Imagine you are assembling a five-person committee to diagnose a rare disease. If you simply hire the five highest-ranked general practitioners in the world, they will likely share the same baseline knowledge and make the same blind-spot errors. They are highly relevant, but highly redundant.

A better strategy is to select the top general practitioner first. For the second seat, you shouldn't look for the second-best general practitioner; instead, you look for a specialist—perhaps a radiologist—who brings entirely new information to the table, even if their standalone diagnostic accuracy is slightly lower than a general practitioner's.

This is exactly how mRMR operates. As formalized by Hanchuan Peng in the seminal paper Minimum Redundancy Feature Selection from Microarray Gene Expression Data, the algorithm balances individual predictive power against collective diversity.

Quantifying Information via Mutual Information

To operationalize relevance and redundancy without assuming linear relationships, mRMR typically relies on Mutual Information (MI). Mutual information measures how much information one variable shares with another, capturing both linear and non-linear dependencies. As noted by Shangce Gao in Mr2DNM: A Novel Mutual Information-Based Dendritic Neuron Model, MI provides a robust metric to quantify exactly how much uncertainty about a target class is reduced by knowing a given feature.

The framework defines two mathematical forces for a subset of features $S$ and a target variable $y$:

  1. Maximum Relevance: Maximize the average mutual information between each selected feature $x_i$ and the target $y$. $$\max D(S, y), \quad D = \frac{1}{|S|} \sum_{x_i \in S} I(x_i; y)$$

  2. Minimum Redundancy: Minimize the average mutual information between all pairs of features within the selected subset $S$. $$\min R(S), \quad R = \frac{1}{|S|^2} \sum_{x_i, x_j \in S} I(x_i; x_j)$$

Balancing the Objectives: MID vs. MIQ

How do we combine these two competing forces into a single optimization score? As discussed by Dong et al. in A naive Bayesian method for eye movement recognition of map linear elements, practitioners generally use one of two criteria to balance relevance and redundancy:

  • Mutual Information Difference (MID): This approach subtracts the redundancy score from the relevance score ($D - R$). It treats both metrics on an absolute scale, prioritizing features where the net gain in unique information is highest.
  • Mutual Information Quotient (MIQ): This approach divides relevance by redundancy ($D / R$). It acts as a relative scaling factor, heavily penalizing features that introduce even a small amount of shared redundancy if their baseline relevance is modest.

The Greedy Selection Process

Finding the absolute global optimum subset of features is an NP-hard problem because evaluating every possible combination of features causes a combinatorial explosion. To make the computation feasible, mRMR uses a greedy ranking algorithm.

  1. Initialization: Evaluate all features individually against the target $y$. Select the single feature with the highest mutual information $I(x_i; y)$ to be the first member of your selected subset $S$.
  2. Iteration: For all remaining features outside of $S$, calculate their relevance to the target and their average redundancy with the features already inside $S$.
  3. Selection: Choose the feature that maximizes the chosen combination metric (MID or MIQ) and move it into $S$.
  4. Termination: Repeat this process until you have reached a predefined number of features or until adding more features yields diminishing returns.

By evaluating redundancy only against the already selected subset rather than all possible combinations, mRMR scales efficiently to massive datasets.

Why It Works Across Domains

The beauty of mRMR is its domain-agnostic nature. Because it relies on information theory rather than a specific underlying machine learning classifier, it acts as a pure "filter-based" preprocessing step. It strips away the noise before the data ever reaches a predictive model.

This versatility is evident across a wide array of complex research domains:

By shifting the focus from individual feature performance to collective feature synergy, mRMR ensures that every variable allowed into a model earns its place by contributing genuinely new information.

Key Foundational Work

GTM-URL contribution to the INTERSPEECH 2009 emotion challenge (2009) — contributed 1 concept elements to this direction.
Minimum Redundancy Feature Selection from Microarray Gene Expression Data (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mrmr_feature_selection_d0cfde4f,
  title = {mRMR feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mrmr-feature-selection/rdr_d0cfde4f2f71440aa0decbe5e1f2d2fe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_mrmr_feature_selection_d0cfde4f_v1,
  title = {mRMR feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mrmr-feature-selection/rdr_d0cfde4f2f71440aa0decbe5e1f2d2fe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake