Abstract
Causal inference research often suffers from fragmented evaluation practices due to the unobservable nature of ground truth counterfactuals. We present the Universal Causal Evaluation Engine (UCEE), a standardized API designed to implement a "common task framework" for causal models. By providing a unified interface for datasets and models, UCEE enables rigorous, reproducible comparisons and automated hyperparameter optimization across diverse causal tasks.
Introduction
In predictive machine learning, evaluation is typically straightforward: models are tested against observable ground truth labels. However, causal inference faces the "Fundamental Problem of Causal Inference"—the inability to observe both treated and untreated outcomes for the same individual simultaneously. Consequently, researchers rely on synthetic or semi-synthetic datasets, often using custom-built data-generating processes and idiosyncratic metrics. This fragmentation hinders "apples-to-apples" comparisons. The Universal Causal Evaluation Engine (UCEE) addresses this by centralizing the evaluation infrastructure, automating the "plumbing" of causal benchmarking.
The UCEE Framework
The UCEE provides a standardized pipeline for evaluating causal models. It adopts a scikit-learn-style interface, where new models inherit from an abstract superclass requiring .fit(X, Y, T) and .predict(X, T) methods. This design ensures familiarity for machine learning practitioners and simplifies the integration of new algorithms.
Key features of the engine include:
- Automated Hyperparameter Tuning: Built-in support for $k$-fold and leave-one-out cross-validation ensures models are evaluated at their optimal settings.
- Standardized Evaluation Loop: The engine manages training, tuning, and testing across multiple dataset realizations, storing results in a centralized database.
- Extensibility: The framework supports the addition of new datasets, models, and evaluation metrics, such as sensitivity analyses.
Benchmark Datasets
The API includes four initial benchmark datasets:
- IHDP (Infant Health and Development Program): A semi-synthetic dataset derived from a randomized experiment, used to evaluate selection bias handling.
- Jobs (LaLonde): A classic benchmark for job training programs with binary unemployment outcomes.
- Voting: A study on the effect of get-out-the-vote calls with an injected synthetic treatment effect.
- Twins: Based on real twin birth data, simulating counterfactuals to study mortality outcomes.
Results and Analysis
Initial evaluations using UCEE reveal significant insights into model performance and robustness. On the IHDP dataset, the T-Learner and OLS-Separate models demonstrated superior precision, achieving the lowest Precision in Estimation of Heterogeneous Effects (PEHE) and Root Mean Squared Error of the Individual Treatment Effect (RMSE ITE).
The results highlight that no single model serves as a "silver bullet." While OLS-Separate performs well on IHDP, it is surpassed by S-Learner and T-Learner on the Jobs and Voting datasets. Furthermore, UCEE facilitates "parameter sweeps" to assess robustness. For instance, testing on the Twins dataset showed that as "proxy noise" (data uncertainty) increases, model errors tend to converge, though certain learners like the T-Learner maintain a performance advantage longer.
Discussion and Future Work
While UCEE currently operates as a local service, future development aims to create a hosted platform for community-driven leaderboards. Current limitations include a primary focus on the Potential Outcomes framework, with plans to incorporate Structural Causal Models (SCMs) and Directed Acyclic Graphs (DAGs). The success of UCEE depends on broad community adoption to establish it as the standard for causal inference evaluation.
Summary updated