Causal Feature Selection for Robust Prediction

17 concepts17 papers1 proposal

Standard machine learning models operate on a simple premise: maximize predictive accuracy by finding patterns in data. If a feature correlates strongly with a target label within a training dataset, Empirical Risk Minimization (ERM) will leverage it.

However, this creates a fundamental vulnerability known as the causal-predictive gap. Statistical correlation does not distinguish between a feature that causes an outcome, a feature that is a downstream effect of the outcome, and a feature that is merely a confounder introduced by the environment. When a model is deployed in a new environment, these non-causal correlations often collapse, causing the model's performance to degrade.

Causal feature selection addresses this vulnerability by shifting the objective from maximizing raw predictive accuracy to identifying the underlying structural mechanisms that generate the data.

The Anatomy of Spuriousness

To understand why causal feature selection works, consider a target variable $Y$ (e.g., a disease) surrounded by three distinct classes of predictive features:

  1. Direct Causes (Parents): Features that physically generate or influence $Y$ (e.g., a genetic mutation).
  2. Downstream Effects (Children): Features generated or influenced by $Y$ (e.g., a clinical symptom).
  3. Confounders or Spouses: Environmental variables that share a common cause with $Y$, or variables that share a common child with $Y$ (e.g., hospital-specific diagnostic equipment).

In a static training dataset, all three categories can be highly predictive of $Y$. However, as Bontempi et al. demonstrate in Causal filter selection in microarray data, selecting downstream effects or bystander variables rather than direct causes leads to fragile models. If an intervention alters the environment, or if the model is tested out-of-distribution, the statistical link between $Y$ and its non-causal features breaks.

This misalignment is particularly dangerous in causal inference and uplift modeling. As Tran et al. show in Improving Data-driven Heterogeneous Treatment Effect Estimation Under Structure Uncertainty, standard feature selection methods often select "bad controls" (such as mediators), which introduces severe bias and invalidates the causal interpretation of the model's predictions. Sun et al. similarly note in Robustness-enhanced Uplift Modeling with Adversarial Feature Desensitization that standard criteria fail to isolate the true drivers of a treatment effect, instead latching onto features that merely correlate with the baseline outcome.

Isolating the Markov Blanket

The mathematical foundation for solving this problem relies on the Markov Blanket—the minimal set of features consisting of a target’s parents, children, and spouses. Conditioned on its Markov Blanket, a target variable becomes statistically independent of all other variables in the system.

   [Confounder]
    /        \
   v          v
[Parent] --> [ Target Y ] --> [Child]
              ^       ^
             /         \
         [Spouse] --> [Shared Child]

By identifying this local causal structure, algorithms can systematically filter out irrelevant environmental noise. Yin et al. exploit this property in Integrating Markov Blanket Discovery Into Causal Representation Learning for Domain Generalization. By discovering the Markov Blanket within a latent representation space, their framework guarantees that the learned predictors remain invariant across shifting domains while retaining maximum information about the target.

To isolate these causal structures from high-dimensional data without knowing the full causal graph in advance, researchers use information-theoretic filters. The min-Interaction Max-Relevance (mIMR) framework introduced by Bontempi et al. analyzes three-way interactions among variables to prioritize direct causes and eliminate redundant downstream effects.

When dealing with complex systems, leveraging multiple targets can further clarify these boundaries. Haibe-Kains et al. extended this approach to multi-input multi-output (MIMO) settings in Multiple-input multiple-output causal strategies for gene selection, proving that evaluating features against secondary phenotypes helps distinguish primary causal drivers from passive biological bystanders.

Neutralizing Confounders and Irrelevant Covariates

When confounders are dense and affect the entire data-generating process, simple filtering is insufficient. Instead, models must be designed to neutralize the confounding background signal directly.

One mechanism is to explicitly partition the model's capacity. Wenzel et al. implement this in Sparse probit linear mixed model by modeling population-wide confounding as a dense, collective effect. By capturing background correlations in a dedicated component, the model frees its sparse regression weights to focus exclusively on true, localized causal signals.

In observational studies and matching frameworks, treating all covariates as equally important introduces significant noise. Roy et al. address this in FLAME: A Fast Large-scale Almost Matching Exactly Approach to Causal Inference by dynamically learning which covariates are truly predictive of the outcome. This prevents irrelevant variables from degrading match quality and introducing bias into treatment effect estimates.

This structural awareness is equally vital across multi-label and time-series domains:

Enforcing Causal Focus in Deep Architectures

The historical roots of this direction trace back to early symbolic AI. Pazzani et al. developed the OCCAM system to integrate prior causal theories with empirical learning, demonstrating that prioritizing underlying mechanisms prevents models from being misled by coincidental feature co-occurrences (The Role of Prior Causal Theories in Generalization).

In modern deep learning, where massive overparameterized models easily exploit surface-level shortcuts, this principle is enforced through counterfactual reasoning and targeted interventions.

In natural language processing, models often latch onto non-causal tokens (such as associating a specific director's name with a positive movie review). To counter this, Choi et al. developed the Causally Contrastive Learning (C2L) framework in C2L: Causally Contrastive Learning for Robust Text Classification. By generating counterfactual examples that alter spurious context while keeping causal words intact, the framework forces text classifiers to focus strictly on the true linguistic drivers of a label.

For vision-language models, Li et al. introduced the JSCPT framework in Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning. This method schedules "causal prompts" to dynamically filter out misleading background elements, ensuring that models like CLIP focus on core object features that remain consistent across varying environments.

This shift fundamentally changes model interpretability and architecture:

  • Faithful Explanations: Rather than relying on post-hoc statistical correlations to explain a model's output, Panda et al. argue for instance-wise causal feature selection in Instance-Wise Causal Feature Selection for Model Interpretation. Optimizing directly for causal influence yields sparser, more faithful explanations that ignore shortcuts.
  • Counterfactual Bottlenecks: Dominici et al. design Counterfactual Concept Bottleneck Models (CF-CBMs) in Counterfactual Concept Bottleneck Models. By jointly training a concept bottleneck with a counterfactual generator, they ensure that the internal representations are true causal drivers of the prediction rather than environmental shortcuts.

As surveyed by Liu et al. in Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey, embedding these causal intervention strategies directly into large language models is becoming essential. Forcing models to prioritize causal features over surface-level token patterns shields them from adversarial attacks and prompt injections, providing a mathematically grounded path toward robust generalization.

Key Foundational Work

The Role of Prior Causal Theories in Generalization (1986) — contributed 1 concept elements to this direction.
Causal filter selection in microarray data (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_feature_selection_spurious_correl_577b2bf6,
  title = {causal feature selection (spurious correlation mitigation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-feature-selection-spurious-correlation-mitigation/rdr_577b2bf67acd46058f53bd081770a9ac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_feature_selection_spurious_correl_577b2bf6_v1,
  title = {causal feature selection (spurious correlation mitigation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-feature-selection-spurious-correlation-mitigation/rdr_577b2bf67acd46058f53bd081770a9ac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake