In standard supervised learning, we evaluate a model by comparing its predictions to observed labels. If we want to know if a cat-classifier works, we show it a new picture of a cat and see if it agrees with the ground truth. But in causal inference, the "ground truth" is a ghost.
When we estimate an Individual Treatment Effect (ITE)—the difference between what happens if a patient takes a drug versus if they don't—we face the Fundamental Problem of Causal Inference: we only ever observe one of those two outcomes. We can never see the counterfactual. This creates a unique crisis for machine learning: how do you perform cross-validation, tune hyperparameters, or select the best architecture when you don't have labels for the quantity you are trying to predict?
The Factual-Counterfactual Gap
The most intuitive way to validate a causal model is to see how well it predicts the outcomes we did observe (the factual outcomes). However, as Uri Shalit, Fredrik Johansson, and David Sontag demonstrate in Estimating individual treatment effect: generalization bounds and algorithms, a model that is excellent at predicting factual outcomes can be disastrous at estimating treatment effects.
This happens because of selection bias. If doctors only give a drug to the healthiest patients, a model might learn that "the drug causes health," when in reality, it's just picking up on the pre-existing health of the treated group. Standard cross-validation on factual data rewards the model for memorizing these correlations rather than learning the true causal contrast. As Danni Lu and Lawrence Carin argue in Reconsidering Generative Objectives For Counterfactual Reasoning, this lack of a validation target makes it nearly impossible to prevent models from overfitting to the factual observations at the expense of counterfactual accuracy.
Searching for Surrogate Risks
To get around the missing labels, researchers have developed "causal risk" metrics—mathematical proxies that stand in for the unobservable counterfactual error.
One of the most robust approaches comes from Alejandro Schuler and Robert Tibshirani. In General-purpose validation and model selection when estimating individual treatment effects, they propose using the R-learner objective ($\tau\text{-risk}_R$) as a validation score. Instead of looking at raw outcomes, the R-learner looks at the residuals—the difference between the observed outcome and what a baseline model expected to happen. By minimizing the error in these residuals, we can rank different models (like S-learners or T-learners) even without seeing the true effect.
Similarly, Alicia Curth and Mihaela van der Schaar have explored how different "meta-learners" (algorithms that decompose the causal problem into standard regression tasks) perform in finite-sample settings. In Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms, they show that while some models are theoretically "optimal" in the limit of infinite data, simpler architectures often win in practice because they are easier to tune using these proxy metrics.
Reframing Causal Inference as Classification
If the problem is that causal inference doesn't have the "full machine learning toolkit," why not change the problem? David Arbour, Drew Dimmery, and Arjun Sondhi take this route with Permutation Weighting (PW).
In Permutation Weighting, they transform the search for balancing weights into a standard binary classification task. By training a classifier to distinguish between the real data and a "shuffled" version where treatment is randomized, they create a clear objective function. This allows practitioners to use familiar tools like cross-validation and gradient boosting to optimize their causal estimators directly, bypassing the need for specialized causal proxies.
Using Theoretical Bounds as a Compass
When empirical validation fails, we can turn to math. Daniel Csillag and colleagues provide a different path in Generalization Bounds for Causal Regression: Insights, Guarantees and Sensitivity Analysis. They developed tight, finite-sample generalization bounds that act as a "safety rating" for models.
Instead of trying to guess the counterfactual error, these bounds penalize models that are overly sensitive to treatment assignment or have high uncertainty. This allows researchers to select models that are mathematically guaranteed to generalize better to unobserved counterfactuals, providing a rigorous foundation for model selection in high-stakes fields like medicine or policy-making where "guessing" is not an option.
Summary of the Direction
The research across these papers converges on a single realization: causal model selection is a distinct discipline from predictive model selection. Whether through surrogate risks like the R-learner, reframing the problem as classification via Permutation Weighting, or using generalization bounds to penalize bias, the goal is to build a "validation set" for a world where the most important data points are fundamentally invisible.