Does Predictive Performance Translate to Causal Performance? Evaluating tabular foundation models as nuisance learners for causal effect estimation

Andrew R Weckstein, Issa J Dahabreh, Marinka Zitnik, Sonia Hernández-Díaz, Sebastian SchneeweissarXiv 2026 · arXiv

TL;DR

Tabular foundation models (PFNs) like TabPFN aren't just for prediction—they are remarkably effective "nuisance learners" for causal inference. This paper demonstrates that PFNs consistently outperform traditional machine learning ensembles in complex causal estimation tasks, providing more accurate treatment effects and more reliable confidence intervals with minimal tuning.

What Problem Does This Paper Address?

Estimating a causal effect (like the impact of a new drug) often requires modeling "nuisance functions": the propensity score (the probability of receiving treatment) and the outcome regression (the expected outcome given covariates and treatment).

While researchers often use flexible machine learning (ML) tools like Random Forests or "Super Learner" ensembles for this, these methods come with trade-offs. They can be sensitive to overfitting, require extensive hyperparameter tuning, and often struggle to provide valid uncertainty quantification (confidence intervals) in complex, non-linear settings. This paper investigates whether a new class of Prior-data Fitted Networks (PFNs)—models pre-trained on millions of synthetic datasets—can solve these issues "out-of-the-box."

Methods

The authors evaluate PFNs (specifically TabPFN and Mitra) within the framework of doubly robust estimators: Augmented Inverse Probability Weighting (AIPW) and Targeted Minimum Loss-based Estimation (TMLE).

The core innovation is replacing traditional "fit-then-predict" models with PFNs that use in-context learning. As shown in Figure 1, these models don't update their weights on your specific dataset; instead, they take your data as "context" and generate predictions in a single forward pass.

Overview of PFN pre-training and in-context learning

The study compares PFNs against a gauntlet of baselines:

  • Parametric models: Standard GLMs.
  • Standalone ML: Random Forest, XGBoost, Highly Adaptive Lasso (HAL).
  • Ensembles: Three variations of "Super Learner" (stacking multiple models).
  • Cross-fitting: All models were tested with and without sample splitting to control for overfitting.

Datasets

The researchers used two distinct simulation setups to stress-test the models:

  1. Simulation 1 (Fully Synthetic): Based on the classic Kang & Schafer design, featuring four confounders. It includes a "Simple" scenario (linear relationships) and a "Complex" scenario (non-linear transformations of the covariates).
  2. Simulation 2 (Plasmode): A more realistic setup using real-world covariate data from the Collaborative Perinatal Project (80 covariate levels). It covers 44 different data-generating processes (DGPs) with varying levels of non-linearity, confounding, and treatment effect heterogeneity.

Key Insights and Ideas

  • Robustness is the Killer App: Most models face a trade-off: they are either good at simple parametric data or good at complex non-linear data. PFNs were the only learners that performed near the top in both regimes.
  • The "Memorization" Trap: At very small sample sizes ($n=200$), PFNs can overfit the propensity score by "memorizing" treatment labels (achieving an AUC of ~1.0). In these cases, cross-fitting is essential to restore performance.
  • Superior Uncertainty Quantification: In complex settings, as sample size increases, PFNs were the only models where confidence interval coverage actually approached the nominal 95%. For other models, coverage often worsened with more data because their bias didn't shrink as fast as their sampling error.
  • Implicit Model Selection: Because PFNs are pre-trained on a vast "prior" of diverse causal structures, they effectively perform model selection and averaging instantly during the forward pass, amortizing the cost of what would usually require expensive cross-validation.

Key Results

  • Complex Scenario Dominance: In the complex synthetic setting, TabPFN achieved <10% bias and ~90% coverage. In contrast, the best non-PFN learners (Super Learners) had 15-30% bias and coverage as low as 25-66%.
  • Plasmode Performance: In the large-scale Simulation 2, TabPFN obtained the lowest average error (RMSE) in 100% of the Average Treatment Effect (ATE) settings and 86% of the Average Treatment Effect on the Treated (ATT) settings.
Performance comparison in Simulation 1
  • Efficiency: TabPFN is fast. On a GPU, it can handle nuisance estimation in about 1.2 seconds, making it significantly faster than complex Super Learner ensembles that take minutes to tune and fit.

Limitations and Future Work

While the results are promising, the authors note several boundaries:

  • Scope: The study focused on continuous outcomes and binary treatments. Performance on categorical outcomes or continuous treatments remains to be seen.
  • Identifiability: The simulations assumed no unmeasured confounding. Real-world performance in "dirty" data with mediators or colliders needs further study.
  • Scaling: While PFNs are fast, current architectures may face memory constraints with extremely large datasets or very high-dimensional feature spaces ($p > 1000$).
  • Future Directions: The authors suggest exploring PFNs specifically trained for causal tasks (e.g., "CausalPFNs") or using the model's internal Bayesian-inspired distribution for even better uncertainty quantification.

Summary updated

Abstract

Tabular foundation models based on prior-data fitted networks (PFNs) achieve strong predictive performance, but their value for causal inference is unclear. We evaluated whether PFNs, which predict through in-context learning, could match or outperform conventional nuisance learners in supporting accurate causal estimation and reliable uncertainty quantification (inference). Learners modeled propensity scores and outcome regressions within doubly robust estimators of average causal effects, with and without cross-fitting, across synthetic and semi-synthetic settings with varying confounding complexity and effect heterogeneity. Under simple parametric data-generating processes, the PFNs were competitive with correctly specified parametric regressions and ensembles containing them. In more complex settings, they outperformed all conventional learners evaluated, including richly specified Super Learner ensembles, for both accuracy and uncertainty quantification. As sample size increased, only the two PFN models (TabPFN and Mitra) consistently approached nominal confidence interval coverage, whereas under-coverage persisted for conventional learners. In some settings, PFNs achieved near-nominal coverage and well-calibrated standard errors even without cross-fitting. At the smallest sample size (n=200) and in the higher-dimensional semi-synthetic settings, PFNs and other flexible learners exhibited poor coverage without cross-fitting. For PFNs, this appeared to be driven by propensity score overfitting consistent with in-sample memorization of treatment labels. These findings highlight the promise of prior-data fitted networks as in-context nuisance learners for causal estimation, while underscoring the need for further evaluation in challenging settings.
BibTeX
@misc{weckstein2026does,
  author = {Andrew R Weckstein and Issa J Dahabreh and Marinka Zitnik and Sonia Hernández-Díaz and Sebastian Schneeweiss},
  title = {{Does Predictive Performance Translate to Causal Performance? Evaluating tabular foundation models as nuisance learners for causal effect estimation}},
  year = {2026},
  eprint = {2611.11381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11381},
}
Figure