An evaluation framework for comparing causal inference models

Niki Kiriakidou, Christos Diou

Abstract

Evaluating causal inference models is challenging due to the lack of counterfactual ground truth. Standard metrics often rely on mean errors, which can be skewed by outliers and fail to indicate statistical significance. This paper introduces a robust evaluation framework utilizing performance profiles and non-parametric statistical tests to assess model efficiency and robustness. By comparing state-of-the-art neural and tree-based models across semi-synthetic and synthetic datasets, we demonstrate that this framework provides a more comprehensive and reliable assessment of model performance than traditional averages.

Introduction

Causal inference—the estimation of treatment effects on outcomes—is a fundamental task in many scientific domains. However, it is inherently difficult because the counterfactual (what would have happened under a different treatment) is never observed. While modern machine learning models, including neural networks and tree-based ensembles, have shown promise, their evaluation remains a bottleneck. Standard practices typically involve averaging errors (e.g., ATE or PEHE) across multiple simulations. This approach is flawed as a few extreme outliers can disproportionately affect the mean, and simple averages do not account for statistical significance.

Methodology

To address these limitations, we propose a multi-layered statistical framework:

  1. Performance Profiles: Developed by Dolan and Moré, these plots display the cumulative distribution of a model's performance relative to the best-performing model across all simulations. They allow for a simultaneous assessment of efficiency (how often a model is the best) and robustness (how well it handles the most difficult scenarios).
  2. Friedman Test: A non-parametric test used to rank multiple models across different simulations without assuming normality in the data distribution.
  3. Bergmann-Hommel Post-hoc Test: This test facilitates pairwise comparisons between models, calculating Adjusted P-Values (APVs) to identify statistically significant differences while controlling for the family-wise error rate.

We evaluated six models: Dragonnet, TARNet, and NEDnet (neural-based); and R-Forest, C-Forest, and BART (tree-based).

Datasets

The framework was validated on two benchmarks:

  • IHDP Dataset: A semi-synthetic dataset based on a real-world infant health study, featuring 25 variables and 1,000 simulated realizations.
  • Synthetic Dataset: A fully synthetic "toy" dataset designed with hidden confounding to test model limits in controlled, difficult scenarios.

Results and Discussion

The evaluation revealed that model performance is highly context-dependent. On the IHDP dataset, neural networks were dominant. Dragonnet and TARNet showed superior efficiency and robustness for the Average Treatment Effect (ATE) metric.

Performance profiles for ATE on IHDP. Curves higher and to the left indicate better performance.

The statistical analysis further clarified these relationships. The framework generates "outperformance" charts that visualize which models are statistically superior to others, providing a clear hierarchy of performance.

Summary of which models statistically outperform others on IHDP (ATE).

In contrast, the Synthetic dataset favored tree-based models. R-Forest performed best for population effects (ATE), while C-Forest excelled in individual effect estimation (PEHE), being the top performer in 80% of simulations. These findings underscore that "averages lie" and that a model with a lower average error may not necessarily be the most robust or statistically superior across all cases.

Conclusion

This research demonstrates that standard mean-based evaluations are insufficient for comparing causal inference models. The proposed framework, incorporating performance profiles and non-parametric tests, offers a more rigorous and informative approach. Future work will extend this evaluation to a broader range of real-world datasets and increasingly complex causal architectures.

Summary updated

BibTeX
@inproceedings{kiriakidou2022evaluation,
  author = {Niki Kiriakidou and Christos Diou},
  title = {{An evaluation framework for comparing causal inference models}},
  booktitle = {SETN},
  year = {2022},
  url = {https://openreview.net/forum?id=6LdhcfmGXwS},
}
Figure