TL;DR
This paper introduces "joint learning" for Causal Prior-data Fitted Networks (PFNs), a class of models trained on synthetic data to perform causal inference. By training these models to predict both factual and counterfactual outcomes simultaneously, the authors capture the inherent dependence between these "alternative worlds," leading to significantly more accurate treatment effect estimates and better-calibrated uncertainty quantification.
What Problem Does This Paper Address?
In the real world, causal inference faces a fundamental hurdle: we only ever observe one outcome for a specific individual—the "factual" one. We never see the "counterfactual"—what would have happened if that same person had received a different treatment. Because we can't see both, we can't directly observe the dependence between these two potential outcomes.
Recent research has leveraged Prior-data Fitted Networks (PFNs)—transformers pre-trained on massive amounts of synthetic data—to estimate causal effects. Because the training data is synthetic, the ground truth for both worlds is available. However, previous causal PFNs still restricted their training to one outcome per unit to mimic real-world data constraints. This forces the models to assume that factual and counterfactual outcomes are independent, which is rarely true in practice and leads to poor estimates of individual treatment effects (ITE) and their associated uncertainty.
Methods
The authors propose a Joint Learning (JT) framework that breaks the real-world constraint during the pre-training phase:
- Joint Distribution Learning: Instead of predicting a single outcome, the PFN is trained to output the joint posterior predictive distribution $p(y_{do(0)}, y_{do(1)} | x, D_{obs})$. This allows the model to learn the "cross-world correlation"—how the two potential outcomes move together.
- 2D Output Layer: To represent this joint distribution, the model uses a 2D grid (histogram) over the two outcomes, supplemented by tail parameters to handle unbounded values.
- ITE and Response Densities: From this joint distribution, the model can directly derive the density for the Individual Treatment Effect (the difference between the two outcomes) and the response densities for each treatment arm.
- ATE Density via Optimal Transport: To estimate the Average Treatment Effect (ATE) density across a population, the authors aggregate individual ITE densities using the 2-Wasserstein barycenter (averaging quantile functions). They then apply conformal calibration to ensure the resulting population-level uncertainty is statistically sound.
Datasets
The authors evaluated their approach across four diverse benchmarks:
- RealCause: A semi-synthetic benchmark where outcomes are generated independently. This served as a "robustness test" to see if the joint model fails when the independence assumption actually holds.
- ComplexMech: A synthetic benchmark featuring nonlinear causal mechanisms and dependent outcomes.
- Case Studies: A set of six causal scenarios (including confounding and mediation) adopted from previous literature.
- Twins: A real-world dataset of twin births in the US, where the outcomes of twin pairs are treated as factual and counterfactual proxies for one another.
Key Insights and Ideas
- Breaking the "Real-World" Mirror: The most significant insight is that while PFNs are applied to real-world data, they shouldn't be trained like they are limited by real-world constraints. If the synthetic prior contains the joint ground truth, the model should see it.
- Statistical Efficiency: Learning from both outcomes of the same unit is more efficient. When outcomes are positively correlated (as is common), the shared noise cancels out in the difference, allowing the model to learn the treatment effect from fewer samples.
- Distributional Causal Inference: Most causal methods provide only a point estimate (a single number). This approach provides full probability densities, which is vital for high-stakes decision-making (e.g., "What is the probability this drug will actually help this patient?").
Key Results
- Improved Accuracy: The Jointly Trained (JT) models significantly outperformed baselines. On the case studies, they reduced the error in treatment effect estimation ($\sqrt{PEHE}$) by 40–62% for Do-PFN and 11–51% for UWYK variants.
- Superior Uncertainty Calibration: The JT models achieved better "interval scores" (a metric combining accuracy and the quality of uncertainty bounds) in 42 out of 48 comparisons across different models and benchmarks.
- Real-World Transfer: On the Twins dataset, joint learning reduced the mean squared error of counterfactual predictions by 22–48% compared to standard baselines.
Limitations and Future Work
- Computational Resolution: Representing a 2D joint distribution on a grid is more expensive than a 1D marginal. To keep costs comparable to baselines, the authors used a coarser grid, which can limit the model's representational resolution.
- Prior Sensitivity: Like all PFNs, the performance is tied to the quality of the synthetic prior. If the prior doesn't reflect the types of dependencies found in the target real-world data, the gains may be limited.
- Direct ATE Modeling: The current ATE density is an approximation via optimal transport. Future work could explore learning the ATE density directly or more explicitly modeling the dependence between different units in a dataset.
Summary updated