TL;DR
Standard evaluations of causal inference models often rely on simple mean error metrics that can be easily skewed by a few difficult outliers. This paper proposes a more rigorous evaluation framework using performance profiles and non-parametric statistical tests to provide concrete evidence of model superiority and statistical significance.
What Problem Does This Paper Address?
Causal inference—estimating the effect of a treatment (like a drug or a policy) on an outcome—is notoriously difficult because we never observe the "counterfactual." We see what happened to a patient who took a drug, but we can't see what would have happened to that exact same patient if they hadn't.
In recent years, many machine learning models (neural networks and tree-based ensembles) have been developed to tackle this. However, the community typically evaluates them by averaging errors across many simulations (using metrics like $| \epsilon_{ATE} |$ for population effects and $PEHE$ for individual effects). The authors argue this is flawed: a small number of extremely difficult simulations can dominate the average, leading to misleading conclusions. Furthermore, simple averages don't tell us if the difference between Model A and Model B is statistically significant or just a result of random noise.
Methods
The authors introduce a multi-layered statistical framework to complement standard error metrics:
- Performance Profiles (Dolan and Moré): Instead of a single number, these plots show the cumulative distribution of a model's performance relative to the best-performing model across all simulations. They reveal both efficiency (how often a model is the best) and robustness (how well it performs on the hardest problems).
- Friedman Test: A non-parametric test used to rank multiple models across different simulations without assuming the data follows a normal distribution.
- Bergmann-Hommel Post-hoc Test: This provides pairwise comparisons between models. It calculates "Adjusted P-Values" (APVs) to determine if one model statistically outperforms another while controlling for the increased risk of false positives when making multiple comparisons.
The study compares six state-of-the-art models: three neural-based (Dragonnet, TARNet, NEDnet) and three tree-based (R-Forest, C-Forest, BART).
Datasets
The framework was tested on two distinct types of data:
- IHDP Dataset: A semi-synthetic dataset based on a real-world study of infant health. It contains 25 features and 1,000 different simulated realizations, providing a robust benchmark for "real-ish" data.
- Synthetic Dataset: A fully synthetic "toy" dataset designed with hidden confounding variables. This is used to push models to their limits in controlled, difficult scenarios.
Key Insights and Ideas
- Averages Lie: The authors demonstrate that a model might have a lower average error but actually be less robust than its competitors. Performance profiles expose this by showing the full distribution of results.
- No Universal Winner: The "best" model depends heavily on the dataset and the specific metric. Neural models like Dragonnet dominated the IHDP dataset, but tree-based models like C-Forest and BART were superior on the Synthetic dataset for individual effect estimation ($PEHE$).
- Visualizing Superiority: The framework allows for "outperformance" charts that clearly visualize which models are statistically better than others, moving beyond the ambiguity of overlapping error bars.
Key Results
On the IHDP dataset, neural networks were the clear winners. Dragonnet and TARNet exhibited the best efficiency and robustness.
The performance profiles for IHDP (Figure 1) show Dragonnet leading the pack, being the "best" solver in 23% of simulations.
The authors' statistical analysis (Figure 2) provides a definitive hierarchy of which models statistically beat others for the ATE metric on IHDP.
However, the story changed for the Synthetic dataset. Here, tree-based models like R-Forest performed best for population effects (ATE), while C-Forest was the standout for individual effects (PEHE), solving 80% of simulations with the lowest error.
Limitations and Future Work
The authors acknowledge that their evaluation is limited to two datasets. While IHDP is a standard benchmark, it is still semi-synthetic. Future work aims to apply this framework to more diverse real-world datasets and more complex causal models. They also suggest investigating why certain models are "dominated" by specific types of problems, which could lead to more specialized model architectures for different data characteristics.
Summary updated