How to select predictive models for decision-making or causal inference

Matthieu Doutreligne, Gaël VaroquauxGigaScience 2025 · DOI

TL;DR

This paper investigates how to choose the best predictive model for making decisions or inferring causal effects, finding that standard prediction metrics are insufficient. They propose and validate a new metric, the R-risk, which better captures causal inference quality and leads to more trustworthy decision-making models.

What Problem Does This Paper Address?

In fields like medicine, predictive models are increasingly used not just to forecast outcomes but to guide decisions about interventions (e.g., whether a patient will benefit from a specific treatment). This requires moving beyond simply predicting the outcome accurately on average. The challenge is that a model that predicts well for the general population might not accurately estimate the effect of an intervention for an individual, especially when factors like confounding (where baseline characteristics influence both treatment assignment and outcome) are present. Standard model selection methods, which focus on minimizing prediction error (like Mean Squared Error or R-squared), can fail to identify models that provide reliable causal estimates. This paper aims to bridge this gap by identifying and evaluating model selection procedures that are appropriate for causal inference tasks.

Methods

The authors explore various model selection procedures, moving beyond standard cross-validation. They focus on "causal risks" that attempt to estimate the quality of causal predictions directly. A key concept is the "outcome model" or "g-computation" framework, which uses models to estimate potential outcomes under different treatment scenarios.

They introduce and analyze several "feasible error risks" (metrics that can be computed from observed data) and compare them to an "oracle t-risk" (the ideal, but uncomputable, measure of causal inference quality). The R-risk is a central focus, presented as a reweighted version of the t-risk that accounts for overlap in treatment assignment and uses nuisance models (like propensity scores and outcome models) to estimate causal effects. They also investigate the use of flexible "stacked" nuisance estimators, which combine multiple machine learning models.

The model selection procedure involves:

  1. Prefitting nuisance estimators: Learning models for quantities like propensity scores and conditional mean outcomes on a training set.
  2. Fitting candidate outcome models: Training various predictive models on the same training set.
  3. Evaluating candidates: Using a chosen causal risk metric (like R-risk) on a separate test set to score each candidate model.
  4. Selecting the best model: Choosing the model with the lowest causal risk.

Datasets

The study utilizes a combination of simulated and real-world datasets:

  • Caussim: A family of extensive simulation settings designed to evaluate the impact of overlap parameters, covariate distributions, and functional links between covariates, outcome, and treatment.
  • ACIC 2016: A dataset based on the Collaborative Perinatal Project, a randomized controlled trial, augmented with synthetic causal effects.
  • ACIC 2018: A dataset based on the Linked Births and Infant Deaths Database, also augmented with synthetic causal effects.
  • Twins: A dataset based on real data on twin births and mortality rates, augmented with simulated treatment allocations.

The authors provide code and data for replication, including a new benchmark dataset generator called "Caussim."

Key Insights and Ideas

  • Standard prediction metrics fail for causal inference: Models that perform best on standard metrics like R-squared can be poor for estimating causal effects, especially in the presence of confounding or poor covariate overlap.
  • The R-risk is a superior metric: The R-risk, which incorporates estimates of nuisance models (propensity scores and outcome models), consistently outperforms other feasible metrics in selecting models that yield good causal estimates.
  • Nuisance models are crucial but manageable: While estimating nuisance models adds complexity, flexible "stacked" estimators (combining multiple models) are effective and can be trained on the same data as the outcome models without significant performance degradation.
  • Overlap matters: Model selection performance degrades significantly when there is low overlap between treated and control groups, highlighting the importance of this assumption in causal inference.
  • Flexibility in nuisance estimation is key: Using adaptive models like stacked estimators for nuisances is sufficient for good model selection, even when simpler models might seem appropriate.

Key Results

  • The R-risk metric consistently showed the best agreement with the oracle t-risk (the ideal measure of causal inference quality) across various datasets and overlap conditions.
  • Using the R-risk led to selecting models with only a ~1% higher causal risk than the best possible model in strong overlap scenarios, compared to ~10% excess risk for standard metrics like MSE (u-risk) or propensity-weighted MSE (µ-riskIPW).
  • Model selection becomes considerably harder with lower population overlap, with Kendall's tau correlation to the oracle t-risk dropping significantly.
  • Estimating nuisance models on the same data as the outcome models (shared sets) yielded similar results to using separate datasets for nuisances (separated sets), suggesting practical efficiency gains.

Limitations and Future Work

  • Computational cost: While the R-risk is effective, it requires a more complex procedure than standard cross-validation, potentially increasing computational burden.
  • Specific scenarios: The paper notes that in situations with very large causal effects relative to baseline variation, other metrics like µ-riskIPW might perform slightly better.
  • Future directions: The authors provide code to facilitate the use of their proposed model selection procedure, opening the door for evaluating a wider range of predictive models, including complex ones like deep neural networks and language models, for causal inference tasks. They also highlight the need for continued research into robust model selection in challenging scenarios, such as low overlap.

Summary updated

Abstract

When predictive models are used to explain the effect of an intervention, they must be evaluated with different procedures than standard predictive settings, using the $R\text{-risk}$ from causal inference.
BibTeX
@inproceedings{doutreligne2025how,
  author = {Matthieu Doutreligne and Gaël Varoquaux},
  title = {{How to select predictive models for decision-making or causal inference}},
  booktitle = {GigaScience},
  year = {2025},
  doi = {10.1093/gigascience/giaf016},
}
Figure