When applying machine learning to causal inference, a seductive intuition often takes hold: the better a model predicts the observed data, the better it will estimate a causal effect.
To isolate a causal relationship or a specific statistical parameter, researchers frequently employ machine learning models to estimate intermediate "nuisance functions"—such as predicting an outcome from covariates or predicting treatment assignments (propensity scores). The standard practice is to tune these nuisance models using cross-validation to minimize global prediction errors like Mean Squared Error (MSE).
However, a growing body of research demonstrates that this approach is fundamentally flawed. As Alan Mishler, Matteo Bonvini, and Edward H. Kennedy argue in their Comment on "Statistical Modeling: The Two Cultures" by Leo Breiman, the bias-variance tradeoff that minimizes global prediction error is often highly suboptimal for estimating a targeted causal effect. Optimizing purely for predictive accuracy introduces systematic biases that can severely degrade the validity of subsequent statistical inferences.
Why Global Prediction Blinds Models to Causal Signals
To understand why prediction-optimal models fail at inference, we must look at how standard machine learning objectives treat data. A global loss function, such as MSE, treats every prediction error equally across the entire distribution. It does not know or care that the researcher's ultimate goal is to isolate a specific contrast—the difference between a treatment outcome and a counterfactual control outcome.
This misalignment creates three distinct structural blindspots:
1. The Contrast Problem
In PairNet: Training with Observed Pairs to Estimate Individual Treatment Effect, Lokesh Nagalapatti et al. point out that models focused solely on minimizing factual outcome errors fail to prioritize the difference between treatment outcomes. A model can achieve exceptional global predictive accuracy by fitting dominant, non-causal features of the data while completely misestimating or flattening the subtle variation driven by the treatment itself. As Claudia Shi, David Blei, and Victor Veitch demonstrate in Adapting Neural Networks for the Estimation of Treatment Effects, a model optimized for causal tasks may actually exhibit higher global MSE on outcomes while providing far more reliable treatment effect estimates, because it intentionally prioritizes confounder-related features over predictive noise.
2. The Confounder Exploitation Trap
Standard cross-validation rewards models for exploiting any statistical regularity that improves prediction on the observed data distribution. In Causal Regularization, Dominik Janzing demonstrates that this property causes cross-validation to actively select models that leverage unobserved confounding variables as predictive shortcuts. Because the objective is predictive performance rather than structural fidelity, the training process under-regularizes non-causal paths, embedding confounding bias directly into the model's architecture.
3. The Over-Adjustment Tension
When predicting treatment assignment (the propensity score), over-optimizing for predictive accuracy can be actively harmful. In Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings, Linsen Li, Aron Culotta, and Nicholas Mattei highlight a fundamental tension: if a model predicts treatment assignment too perfectly, it can lead to over-adjustment. This strips away the overlapping variance between the treatment and covariates, making it impossible to isolate the genuine causal signal from the background noise.
The Mathematics of Misaligned Hyperparameters
The conflict between prediction and inference is not merely conceptual; it manifests mathematically in the selection of hyperparameters, such as regularization penalties and smoothing parameters.
Consider a high-dimensional setting where we use a regularized regression (like Lasso) to control for covariates. Lasso shrinks coefficients toward zero to reduce variance and improve prediction. If a covariate is a weak predictor of the outcome but a strong confounder of the treatment, a prediction-optimal tuning strategy will shrink its coefficient to zero to minimize global MSE. As Susan Athey et al. note in Estimating Average Treatment Effects: Supplementary Analyses and Remaining Challenges and Philipp Bach et al. echo in DoubleML: An Object-Oriented Implementation of Double Machine Learning in R, this regularization inherent to predictive ML introduces severe omitted variable bias into the final causal estimate.
[Standard Cross-Validation] ──> Minimizes Global MSE ──> Drops Weak Predictors ──> Omitted Variable Bias
│
(Even if strong confounder)
In Optimal Nuisance Function Tuning for Estimating a Doubly Robust Functional under Proportional Asymptotics, Sean McGrath et al. formally prove that under high-dimensional proportional asymptotics, the hyperparameter values that minimize prediction error for intermediate nuisance models are mathematically distinct from those that minimize the variance of the final target estimate.
A similar misalignment occurs in nonparametric smoothing. Yun Yang et al. demonstrate in Non-asymptotic Analysis for Nonparametric Testing that smoothing parameters optimized via cross-validation for prediction are systematically suboptimal for maximizing the power of a subsequent hypothesis test. This occurs because causal risk and statistical risk operate under entirely different bias-variance tradeoffs—a reality further explored by Leena C. Vankadara et al. in Interpolation and Regularization for Causal Learning, who show that causal learning requires significantly stronger or structurally different regularization to mitigate confounding bias than standard statistical learning.
Realigning Objectives: Targeted and Debiased Estimation
To overcome this tradeoff, researchers must move away from naive predictive optimization and adopt frameworks that explicitly align the training of nuisance models with the ultimate inferential goal.
Targeted Minimum Loss Estimation (TMLE)
One of the primary solutions is TMLE, a doubly robust framework that updates predictive models to minimize bias in the specific target parameter. In Predicting Is Not Explaining: Targeted Learning of the Dative Alternation, Guillaume Desagulier demonstrates how porting TMLE to fields like corpus linguistics prevents machine learning models from mistaking global predictive accuracy for true scientific explanation.
By optimizing a non-parametric variable importance measure rather than global fit, as detailed by Antoine Chambaz, Pierre Neuvial, and Mark J. van der Laan in Estimation of a non-parametric variable importance measure of a continuous exposure, TMLE forces the optimization process to prioritize the precision of the targeted effect size. This principle is increasingly being integrated directly into machine learning architectures, such as the tree-boosting framework proposed by Niki Kiriakidou, Christos Diou, and Ioannis E. Livieris in C-XGBoost: A tree boosting model for causal effect estimation.
Causal-Aware Hyperparameter Tuning
When using frameworks like Double Machine Learning (DML), relying on default hyperparameters or standard predictive cross-validation remains a major source of error. In Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study, Philipp Bach et al. show that proper tuning of nuisance parameters within cross-fitting folds is a strict prerequisite for valid causal conclusions. They emphasize that hyperparameter optimization must be guided by metrics that monitor how well the model isolates the treatment effect, rather than raw predictive performance alone.
Without these specialized adjustments, models suffer from "causal creep"—a phenomenon documented by Antonius Mulder et al. in Considering causality in normal tissue complication probability model development: a literature review, where models optimized purely for predictive power are inappropriately used to guide clinical interventions without adjusting for the confounding factors that make those recommendations valid. To make machine learning a reliable tool for discovery, the field must continue to design objectives where estimation, not prediction, is the North Star.
Key Foundational Work
Predicting Is Not Explaining: Targeted Learning of the Dative Alternation (2015) — contributed 1 concept elements to this direction.
Estimation of a non-parametric variable importance measure of a continuous exposure (2012) — contributed 1 concept elements to this direction.