Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models

Alexander Lin, Amil Merchant, Suproteem K. Sarkar, Alexander D'Amour

Abstract

Causal inference research often suffers from fragmented evaluation practices due to the unobservable nature of ground truth counterfactuals. We present the Universal Causal Evaluation Engine (UCEE), a standardized API designed to implement a "common task framework" for causal models. By providing a unified interface for datasets and models, UCEE enables rigorous, reproducible comparisons and automated hyperparameter optimization across diverse causal tasks.

Introduction

In predictive machine learning, evaluation is typically straightforward: models are tested against observable ground truth labels. However, causal inference faces the "Fundamental Problem of Causal Inference"—the inability to observe both treated and untreated outcomes for the same individual simultaneously. Consequently, researchers rely on synthetic or semi-synthetic datasets, often using custom-built data-generating processes and idiosyncratic metrics. This fragmentation hinders "apples-to-apples" comparisons. The Universal Causal Evaluation Engine (UCEE) addresses this by centralizing the evaluation infrastructure, automating the "plumbing" of causal benchmarking.

The UCEE Framework

The UCEE provides a standardized pipeline for evaluating causal models. It adopts a scikit-learn-style interface, where new models inherit from an abstract superclass requiring .fit(X, Y, T) and .predict(X, T) methods. This design ensures familiarity for machine learning practitioners and simplifies the integration of new algorithms.

The UCEE architecture standardizes the flow from researchers and datasets through an API evaluation layer to final metrics.

Key features of the engine include:

  • Automated Hyperparameter Tuning: Built-in support for $k$-fold and leave-one-out cross-validation ensures models are evaluated at their optimal settings.
  • Standardized Evaluation Loop: The engine manages training, tuning, and testing across multiple dataset realizations, storing results in a centralized database.
  • Extensibility: The framework supports the addition of new datasets, models, and evaluation metrics, such as sensitivity analyses.

Benchmark Datasets

The API includes four initial benchmark datasets:

  1. IHDP (Infant Health and Development Program): A semi-synthetic dataset derived from a randomized experiment, used to evaluate selection bias handling.
  2. Jobs (LaLonde): A classic benchmark for job training programs with binary unemployment outcomes.
  3. Voting: A study on the effect of get-out-the-vote calls with an injected synthetic treatment effect.
  4. Twins: Based on real twin birth data, simulating counterfactuals to study mortality outcomes.

Results and Analysis

Initial evaluations using UCEE reveal significant insights into model performance and robustness. On the IHDP dataset, the T-Learner and OLS-Separate models demonstrated superior precision, achieving the lowest Precision in Estimation of Heterogeneous Effects (PEHE) and Root Mean Squared Error of the Individual Treatment Effect (RMSE ITE).

Performance comparison of various causal models on the IHDP dataset across PEHE and RMSE ITE metrics.

The results highlight that no single model serves as a "silver bullet." While OLS-Separate performs well on IHDP, it is surpassed by S-Learner and T-Learner on the Jobs and Voting datasets. Furthermore, UCEE facilitates "parameter sweeps" to assess robustness. For instance, testing on the Twins dataset showed that as "proxy noise" (data uncertainty) increases, model errors tend to converge, though certain learners like the T-Learner maintain a performance advantage longer.

Discussion and Future Work

While UCEE currently operates as a local service, future development aims to create a hosted platform for community-driven leaderboards. Current limitations include a primary focus on the Potential Outcomes framework, with plans to incorporate Structural Causal Models (SCMs) and Directed Acyclic Graphs (DAGs). The success of UCEE depends on broad community adoption to establish it as the standard for causal inference evaluation.

Summary updated

Abstract

A major driver in the success of predictive machine learning has been the “common task framework,” where community-wide benchmarks are shared for evaluating new algorithms. This pattern, however, i...
BibTeX
@inproceedings{lin2019universal,
  author = {Alexander Lin and Amil Merchant and Suproteem K. Sarkar and Alexander D'Amour},
  title = {{Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models}},
  booktitle = {CD @ KDD},
  year = {2019},
}
Figure