Counterfactual Cross-Validation: Effective Causal Model Selection from Observational Data

Yuta Saito, Shota Yasui

Abstract

We study the model selection problem in conditional average treatment effect (CATE) prediction. Unlike previous works on this topic, we focus on preserving the rank order of the performance of candidate CATE predictors to enable accurate and stable model selection. To this end, we analyze the model performance ranking problem and formulate guidelines to obtain a better evaluation metric. We then propose a novel metric that can identify the ranking of the performance of CATE predictors with high confidence. Empirical evaluations demonstrate that our metric outperforms existing metrics in both model selection and hyperparameter tuning tasks.

Introduction

In standard machine learning, model evaluation is straightforward: compare predictions to observed labels. However, in causal inference, we aim to predict the Conditional Average Treatment Effect (CATE)—the difference between potential outcomes under treatment and control. Because we only observe one outcome for any individual, the "fundamental problem of causal inference" prevents us from having ground-truth labels for the effect itself. This makes traditional cross-validation impossible. Existing metrics like Inverse Probability Weighting (IPW) or "plug-in" learners often suffer from high variance or instability, potentially leading to the deployment of sub-optimal or harmful models.

Methodology: Ranking-Focused Model Selection

The authors propose Counterfactual Cross-Validation (CF-CV), shifting the focus from absolute error estimation to preserving the rank order of candidate models. The core of CF-CV is a novel "plug-in tau" ($\tilde{\tau}$) based on the Doubly Robust (DR) estimator. The procedure involves:

  1. Learning a Specialized Regression Function ($f$): Training a model using a weighted loss and an Integral Probability Metric (IPM) penalty to ensure generalization across treatment groups despite selection bias in observational data.
  2. Constructing a Stable Metric: Using this regression function within a Doubly Robust framework to create a stable performance estimator.

Theoretically, the authors demonstrate that by minimizing a specific variance bound, they can make model selection significantly more stable across finite samples.

Experimental Evaluation

Evaluation was conducted on the Infant Health Development Program (IHDP) dataset, a semi-synthetic benchmark. Semi-synthetic data is essential here because it provides the "ground truth" CATE needed to verify the accuracy of the validation metric, which is unobservable in purely real-world datasets.

Main Results

CF-CV was compared against common baselines including IPW validation, $\tau$-risk, and simple plug-in validation. As shown in the table below, CF-CV significantly outperforms baselines across all key metrics.

Comparison of CF-CV against baseline metrics on the IHDP dataset.

CF-CV achieved the highest Spearman rank correlation (0.921), indicating that its model rankings most closely match the true performance order. It also demonstrated the lowest mean and worst-case regret (0.066 mean vs. 1.032 for IPW), which is critical for safety-sensitive applications like personalized medicine. When used for hyperparameter tuning, CF-CV improved worst-case NRMSE by 1.4% over the best baseline.

Stability and Hyperparameter Sensitivity

The stability of CF-CV was further tested across varying hyperparameter settings (alpha), which controls the IPM penalty weight during training.

Spearman Rank Correlation and Regret in Model Selection for varying values of alpha.

As illustrated, CF-CV maintains higher rank correlation and lower regret compared to the standard plug-in approach across a wide range of alpha values, demonstrating its robustness for hyperparameter tuning tasks.

Conclusion

CF-CV provides a more reliable framework for causal model selection by prioritizing ranking stability and minimizing finite-sample uncertainty. While it assumes unconfoundedness and introduces some computational overhead, its superior performance in identifying the best-performing models makes it a significant advancement for practical causal inference. Future work may extend these principles to off-policy evaluation and bandit settings.

Summary updated

Abstract

We study the model selection problem in conditional average treatment effect (CATE) prediction. Unlike previous works on this topic, we focus on preserving the rank order of the performance of candidate CATE predictors to enable accurate and stable model selection. To this end, we analyze the model performance ranking problem and formulate guidelines to obtain a better evaluation metric. We then propose a novel metric that can identify the ranking of the performance of CATE predictors with high confidence. Empirical evaluations demonstrate that our metric outperforms existing metrics in both model selection and hyperparameter tuning tasks.
BibTeX
@misc{saito2019counterfactual,
  author = {Yuta Saito and Shota Yasui},
  title = {{Counterfactual Cross-Validation: Effective Causal Model Selection from Observational Data}},
  year = {2019},
  eprint = {1909.05299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.05299},
}
Figure