Causal filter selection in microarray data

Gianluca Bontempi, Patrick E. MeyerICML 2010

In fields like genomics, researchers frequently encounter the "large $p$, small $n$" problem: datasets containing thousands of gene expression levels (features, $p$) but only a few dozen clinical samples ($n$). Standard machine learning filters excel at selecting a subset of genes that predict a disease, but they often fail to distinguish between the causes of a disease and its consequences (effects).

For a biologist or clinician, a gene that is merely a consequence of a disease is a poor therapeutic target, even if it serves as a perfect predictor. Furthermore, standard feature selection methods can be highly unstable, selecting entirely different sets of genes if the training data is slightly perturbed.

To address this, researchers have introduced mIMR (min-Interaction Max-Relevance), a feature selection filter that uses information theory to bridge the gap between predictive accuracy and causal discovery. By prioritizing direct causes and filtering out redundant effects, mIMR creates more stable, accurate, and scientifically meaningful gene expression models.

The Methods: Information Theory as a Causal Compass

The core logic of the mIMR filter relies on Information Theory, specifically leveraging the concept of 3-way Interaction Information to deduce causal structures.

Interaction as a Causal Signal

The paper formalizes how the interaction between two features ($x_1, x_2$) and a target ($y$) reveals the underlying causal graph:

  • Negative Interaction (Synergy): Suggests a "common effect" or v-structure (e.g., $x_1 \to y \leftarrow x_2$). The features provide more information together than they do individually.
  • Positive Interaction (Redundancy): Suggests a "causal chain" or "common cause" (e.g., $y \to x_1 \to x_2$). The features share overlapping information about the target.
Causal patterns and their corresponding information-theoretic signatures

The mIMR Algorithm

The algorithm operates in two primary steps:

  1. Relevance Filtering (Solving the "Spouse" Problem): In causal graphs, a "spouse" is a variable that shares a child with the target but has no direct causal link to it. Spouses can appear highly predictive when the target is known. mIMR first discards any variable with zero marginal mutual information with the target ($I(x_i; y) = 0$), effectively ignoring these predictive decoys.
  2. Incremental Selection: It then selects features that maximize relevance to the target while minimizing interaction with already selected features. This subtle shift from 2-way redundancy (used in older algorithms like mRMR) to 3-way interaction allows mIMR to distinguish between "useful synergy" and "useless redundancy."

To handle the massive dimensionality of microarray data efficiently, the authors utilize a Gaussian assumption to estimate mutual information. This turns complex, computationally heavy integral calculations into simple correlation matrix operations.

Datasets and Experimental Validation

The researchers evaluated mIMR using two distinct approaches to test both real-world applicability and theoretical causal accuracy:

  1. Real-World Benchmarks: The method was tested on 25 public domain microarray datasets (including well-known sets like Gordon, Golub, Alon, and West). These datasets cover various cancers and conditions, with feature counts ranging from 1,000 to over 24,000.
  2. Artificial "Toy" Dataset: To explicitly test causal discovery, they used a synthetic dataset inspired by the LUCAS (Lung Cancer Artificial Study) benchmark. It consists of 11 continuous inputs and one target with known linear causal dependencies.
The synthetic LUCAS-inspired causal graph with 11 inputs and one target

Key Insights and Results

  • Superior Accuracy: Across the 25 real-world microarray datasets, mIMR consistently outperformed or matched state-of-the-art filters like DISR, mRMR, and CMIM in predictive accuracy.
  • High Stability: One of the most significant findings is that causal features are inherently more stable across different subsets of data. Because they represent the underlying mechanism of the system, they do not vanish when the training/test split changes. mIMR proved to be 14 to 21 times more stable than competitors like mRMR and CMIM across cross-validation folds.
  • Causal Accuracy: In the synthetic toy example, mIMR successfully identified the true causal drivers. As the number of samples increased, mIMR consistently ranked the two true direct causes (nodes 1 and 5) higher than other methods, proving its ability to filter out downstream effects and spouses.

Limitations and Future Directions

While highly effective, the current implementation of mIMR has a few limitations. First, it relies on a linearity (Gaussian) assumption for computational speed. While this prevents overfitting in small-sample datasets, it may miss complex, non-linear causal interactions in larger datasets. Second, it faces a three-way limit: the algorithm only considers interactions between pairs of features and the target, ignoring potentially informative higher-order interactions (e.g., 4-way or 5-way) due to computational expense.

Future work will likely explore more robust, non-parametric estimators of mutual information that can maintain mIMR's causal prioritization without being constrained by the Gaussian assumption.

Summary updated

BibTeX
@inproceedings{bontempi2010causal,
  author = {Gianluca Bontempi and Patrick E. Meyer},
  title = {{Causal filter selection in microarray data}},
  booktitle = {International Conference on Machine Learning},
  year = {2010},
}
Figure