An evaluation framework for comparing causal inference models

Niki Kiriakidou, Christos Diou

TL;DR

Standard evaluations of causal inference models often rely on simple mean error metrics that can be easily skewed by a few difficult outliers. This paper proposes a more rigorous evaluation framework using performance profiles and non-parametric statistical tests to provide concrete evidence of model superiority and statistical significance.

What Problem Does This Paper Address?

Causal inference—estimating the effect of a treatment (like a drug or a policy) on an outcome—is notoriously difficult because we never observe the "counterfactual." We see what happened to a patient who took a drug, but we can't see what would have happened to that exact same patient if they hadn't.

In recent years, many machine learning models (neural networks and tree-based ensembles) have been developed to tackle this. However, the community typically evaluates them by averaging errors across many simulations (using metrics like $| \epsilon_{ATE} |$ for population effects and $PEHE$ for individual effects). The authors argue this is flawed: a small number of extremely difficult simulations can dominate the average, leading to misleading conclusions. Furthermore, simple averages don't tell us if the difference between Model A and Model B is statistically significant or just a result of random noise.

Methods

The authors introduce a multi-layered statistical framework to complement standard error metrics:

  1. Performance Profiles (Dolan and Moré): Instead of a single number, these plots show the cumulative distribution of a model's performance relative to the best-performing model across all simulations. They reveal both efficiency (how often a model is the best) and robustness (how well it performs on the hardest problems).
  2. Friedman Test: A non-parametric test used to rank multiple models across different simulations without assuming the data follows a normal distribution.
  3. Bergmann-Hommel Post-hoc Test: This provides pairwise comparisons between models. It calculates "Adjusted P-Values" (APVs) to determine if one model statistically outperforms another while controlling for the increased risk of false positives when making multiple comparisons.

The study compares six state-of-the-art models: three neural-based (Dragonnet, TARNet, NEDnet) and three tree-based (R-Forest, C-Forest, BART).

Datasets

The framework was tested on two distinct types of data:

  • IHDP Dataset: A semi-synthetic dataset based on a real-world study of infant health. It contains 25 features and 1,000 different simulated realizations, providing a robust benchmark for "real-ish" data.
  • Synthetic Dataset: A fully synthetic "toy" dataset designed with hidden confounding variables. This is used to push models to their limits in controlled, difficult scenarios.

Key Insights and Ideas

  • Averages Lie: The authors demonstrate that a model might have a lower average error but actually be less robust than its competitors. Performance profiles expose this by showing the full distribution of results.
  • No Universal Winner: The "best" model depends heavily on the dataset and the specific metric. Neural models like Dragonnet dominated the IHDP dataset, but tree-based models like C-Forest and BART were superior on the Synthetic dataset for individual effect estimation ($PEHE$).
  • Visualizing Superiority: The framework allows for "outperformance" charts that clearly visualize which models are statistically better than others, moving beyond the ambiguity of overlapping error bars.

Key Results

On the IHDP dataset, neural networks were the clear winners. Dragonnet and TARNet exhibited the best efficiency and robustness.

Friedman rankings for IHDP show Dragonnet as the top performer for ATE.

The performance profiles for IHDP (Figure 1) show Dragonnet leading the pack, being the "best" solver in 23% of simulations.

Performance profiles for ATE on IHDP. Curves higher and to the left indicate better performance.

The authors' statistical analysis (Figure 2) provides a definitive hierarchy of which models statistically beat others for the ATE metric on IHDP.

Summary of which models statistically outperform others on IHDP (ATE).

However, the story changed for the Synthetic dataset. Here, tree-based models like R-Forest performed best for population effects (ATE), while C-Forest was the standout for individual effects (PEHE), solving 80% of simulations with the lowest error.

Limitations and Future Work

The authors acknowledge that their evaluation is limited to two datasets. While IHDP is a standard benchmark, it is still semi-synthetic. Future work aims to apply this framework to more diverse real-world datasets and more complex causal models. They also suggest investigating why certain models are "dominated" by specific types of problems, which could lead to more specialized model architectures for different data characteristics.

Summary updated

Abstract

Estimation of causal effects is the core objective of many scientific disciplines. However, it remains a challenging task, especially when the effects are estimated from observational data. Recently, several promising machine learning models have been proposed for causal effect estimation. The evaluation of these models has been based on the mean values of the error of the Average Treatment Effect (ATE) as well as of the Precision in Estimation of Heterogeneous Effect (PEHE). In this paper, we propose to complement the evaluation of causal inference models using concrete statistical evidence, including the performance profiles of Dolan and Moré, as well as non-parametric and post-hoc statistical tests. The main motivation behind this approach is the elimination of the influence of a small number of instances or simulation on the benchmarking process, which in some cases dominate the results. We use the proposed evaluation methodology to compare several state-of-the-art causal effect estimation models.
BibTeX
@inproceedings{kiriakidou2022evaluation,
  author = {Niki Kiriakidou and Christos Diou},
  title = {{An evaluation framework for comparing causal inference models}},
  booktitle = {SETN},
  year = {2022},
  url = {https://openreview.net/forum?id=6LdhcfmGXwS},
}

Loading related content...

Figure