Predictive Performance for Causal Model Selection

This direction is split into other directions: out-of-sample causal tuning. Its last version stays readable and citable here.

Version 1 of 117 papers25 conceptsfrozen 2026-10-04

In standard machine learning, we know if a model is good by checking its predictions on a test set. In causal inference, this is rarely possible. We want to know the "counterfactual"—what would have happened if a patient had received a different drug—but we only ever observe the "factual" outcome. Because we lack ground-truth labels for causal effects or the true underlying graph of relationships, tuning hyperparameters and selecting models has historically relied on theoretical assumptions rather than empirical validation.

A growing body of research suggests a way out of this "instability nightmare": using predictive performance on observed data as a reliable proxy for causal validity. By reframing causal discovery and effect estimation as specialized prediction tasks, researchers can use standard cross-validation to find the most accurate causal models.

Structural Correctness via Markov Blankets

When trying to discover the causal graph (who causes what) in a dataset, there is usually no "answer key" to check against. Konstantina Biza and Sofia Triantafillou address this in Tuning Causal Discovery Algorithms by introducing Out-of-sample Causal Tuning (OCT).

The intuition is that a correctly identified causal structure should yield the most efficient set of features for prediction. In a causal graph, a variable’s "Markov Blanket" (its parents, children, and children's parents) contains all the information needed to predict that variable; everything else is redundant. OCT treats each node in a learned graph as a local predictive model. By measuring how well a node’s identified Markov Blanket predicts its value on held-out data, researchers can rank different algorithms and hyperparameters. If a discovered structure fails to predict unseen data well, the structure itself is likely wrong. This principle is central to the ETIA pipeline, which automates causal discovery by treating structural correctness as a measurable predictive objective.

Causal Weighting as Binary Classification

Estimating the effect of a treatment often requires "balancing" the data so that the treated and control groups look similar. This is usually done by assigning weights to individuals. But how do we know if the weights are correct?

David Arbour, Drew Dimmery, and Arjun Sondhi propose Permutation Weighting, which transforms this into a supervised learning problem. They train a classifier to distinguish between the real observed data and a "permuted" version where the relationship between treatment and covariates is broken.

The breakthrough here is that the causal estimation error is theoretically bounded by the classifier's performance. This means researchers can use standard metrics like Log-Loss or AUC to tune their causal models. If the classifier is good at its job, the resulting weights will be good at balancing the groups, leading to more accurate causal estimates.

Factual Accuracy as a Counterfactual Proxy

If we cannot observe the counterfactual, can we trust a model that is highly accurate on the factual data we do have? Armand Lacombe argues in Alrite that factual prediction accuracy is an effective proxy for counterfactual performance.

This idea is echoed in the work of Susan Athey and Guido Imbens, who introduced Causal Trees in Recursive Partitioning for Heterogeneous Causal Effects. They use an "honest" estimation strategy: splitting the data so that one set is used to discover the structure (where the effects vary) and another is used to estimate the magnitude of those effects. This prevents the model from "p-hacking" or overfitting to noise, ensuring that the discovered patterns are statistically valid.

Similarly, Philipp Bach and Victor Chernozhukov show that monitoring the predictive loss of the outcome variable is essential for Double Machine Learning. Models that minimize predictive error on the outcome typically yield the most unbiased causal estimates, allowing practitioners to choose between simple constant-effect models and complex heterogeneous ones.

Stability through Regularization

Even with good proxies, some causal methods—like Instrumental Variable (IV) regression—are notoriously unstable. Zihao Li and Vasilis Syrgkanis address this in Regularized DeepIV. By applying Tikhonov regularization, they make the causal loss function strongly convex. This mathematical stability is what allows "oracle" model selection to work: it ensures that the model that looks best on validation data is actually the one closest to the true causal function.

This shift toward proxy-based validation—seen also in the work of Seok-Jin Kim on Two-Stage Kernel Ridge Regression—means that causal inference is no longer a "black box" of assumptions. By linking causal validity to out-of-sample predictive power, the field is adopting the same rigorous, data-driven tuning standards that fueled the success of deep learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_predictive_performance_for_causal_model__00c967bd,
  title = {Predictive Performance for Causal Model Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predictive-performance-for-causal-model-selection/rdr_00c967bd5fb94ed3886e917b51b30af8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_predictive_performance_for_causal_model__00c967bd_v1,
  title = {Predictive Performance for Causal Model Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predictive-performance-for-causal-model-selection/rdr_00c967bd5fb94ed3886e917b51b30af8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}