MediPFN: Amortized In-Context Nuisance Learning and Inference for Causal Mediation Analysis

Shi Bo, Jiaming Hu

TL;DR

MediPFN is a new framework that brings the power of "tabular foundation models" to causal mediation analysis, allowing researchers to decompose treatment effects into direct and indirect pathways with unprecedented speed. By using a pre-trained model to learn "nuisance" functions and a specialized head to predict error quantiles, it delivers reliable confidence intervals nearly 60 times faster than traditional bootstrap methods.

What Problem Does This Paper Address?

Causal mediation analysis is a statistical tool used to understand how a treatment works—for instance, does a new drug improve health directly, or does it work by changing a specific biological mediator? To answer this, researchers must estimate several "nuisance functions," such as propensity scores (the probability of receiving treatment) and outcome regressions.

The challenge is twofold:

  1. Complexity: These functions are often non-linear and interact in complex ways. Traditional methods fit them from scratch for every new study, which is inefficient.
  2. Uncertainty: Constructing confidence intervals—essential for knowing if an effect is "real"—usually requires computationally heavy techniques like bootstrapping (repeatedly re-fitting the model on different subsets of data).

The authors aim to "amortize" this process: learning these patterns once across many simulated datasets so that inference on a new, real-world dataset is fast and accurate.

Methods

MediPFN (Mediation Prior-Data Fitted Network) builds on TabPFN, a transformer-based foundation model designed for tabular data. The framework consists of three main stages:

  • Adaptation: The authors take a frozen TabPFN backbone and add a trainable "adaptation head." This head learns to map raw data into the right space for the backbone to predict the specific nuisance functions needed for mediation (e.g., $E[Y|A, M, X]$).
  • Influence-Function Estimation: Instead of a simple "plug-in" estimate, MediPFN uses a cross-fitted influence-function estimator. This approach is "multiply robust," meaning it can still provide accurate results even if some of the underlying nuisance models are slightly off.
  • Amortized Uncertainty Quantification: This is the "secret sauce." Rather than bootstrapping, the authors train a separate error-quantile head. This head is trained on thousands of simulated tables where the true indirect effect is known. It learns to predict the quantiles of the estimator's error directly from the data.
A schematic overview of MediPFN adaptation, uncertainty training, and inference.

The final pipeline is refined through supervised fine-tuning (SFT) on semi-synthetic data and an independent calibration step to ensure the resulting confidence intervals are trustworthy.

Datasets

The authors evaluated MediPFN using a rigorous mix of data:

  • Synthetic Data: 1,152 tables generated from various causal mechanism families. This included "in-distribution" tables and "shift" tables (source and composition shifts) to test generalization.
  • Semi-synthetic Data: They used real-world covariates from eight OpenML datasets (like adult, car, and eucalyptus) and simulated the treatment and outcome relationships. This ensures the model handles the "messiness" of real feature distributions.
  • Stress Tests: 576 tables specifically designed to push the model's limits regarding sample size ($n$ as low as 64), heavy-tailed noise, and mechanism shifts.

Key Insights and Ideas

  • Foundation Models for Causal Inference: The paper demonstrates that the "pre-train on everything" philosophy of foundation models works for causal tasks. TabPFN’s ability to perform in-context learning allows it to adapt to new causal structures without full re-training.
  • Learning the Error Distribution: The most innovative leap is treating uncertainty quantification as a supervised learning problem. By showing the model thousands of "known" causal truths, it learns to "sense" its own estimation error.
  • Efficiency via Amortization: By moving the heavy lifting (training) to an offline phase, the online phase (inference on a new dataset) becomes a simple forward pass through a neural network.

Key Results

  • Superior Coverage: In 95% nominal coverage tests, MediPFN consistently outperformed baselines like Linear SEM and Random Forests. On in-distribution synthetic data, it achieved 94.01% coverage, significantly closer to the target than most competitors.
  • Massive Speedup: In matched comparisons, MediPFN was 59.1 times faster than bootstrapping. This makes it feasible to run complex mediation analyses on large-scale datasets or in automated discovery pipelines where bootstrapping would be a bottleneck.
  • Robustness: Even under "source shift"—where the data-generating mechanism was entirely different from anything seen during training—MediPFN maintained a respectable 92.97% coverage.
Median inference time for MediPFN and bootstrap.

Limitations and Future Work

While powerful, MediPFN relies on standard causal assumptions:

  • No Unmeasured Confounding: It assumes there are no hidden variables influencing both the mediator and the outcome.
  • Simulation Prior: The model's performance is ultimately tied to the diversity of its training simulations. If a real-world dataset falls too far outside the "prior" established during training, accuracy may suffer.

Future directions suggested by the authors include extending the framework to handle instrumental variables, performing sensitivity analysis for unmeasured confounding, and investigating even broader distribution shifts.

Summary updated

Abstract

Causal mediation analysis decomposes a treatment effect into indirect pathways through mediators and direct pathways through other processes. Tabular foundation models perform classification and regression through in-context learning, providing estimates of conditional means and propensity scores used in causal mediation analysis. Constructing confidence intervals from these estimates requires accounting for uncertainty introduced by fitting and combining several nuisance functions. We introduce MediPFN, an amortized inference framework that adapts a pretrained tabular foundation model to nuisance prediction and combines its predictions through a cross-fitted influence-function estimator. A separate head learns quantiles of the estimator's standardized error from simulated tables with known indirect effects. Supervised fine-tuning, symmetrization, and independent calibration then produce a confidence interval. We establish coverage bounds for the resulting intervals under standard assumptions. Across experiments, MediPFN attains coverage closer to the nominal level than several comparators. It also reduces the computational cost of uncertainty quantification, running 59.1 times faster than bootstrap in matched comparisons.
BibTeX
@misc{bo2026medipfn,
  author = {Shi Bo and Jiaming Hu},
  title = {{MediPFN: Amortized In-Context Nuisance Learning and Inference for Causal Mediation Analysis}},
  year = {2026},
  eprint = {2611.13219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13219},
}
Figure