IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark

Fredrik D. Johansson

Introduction

In the world of causal inference, establishing "ground truth" is the ultimate challenge. To determine if a specific intervention—like a job training program—actually works, researchers must estimate the "counterfactual": what would have happened to the exact same person had they not participated. Because we cannot observe two realities simultaneously, the field relies heavily on simulators to test and validate causal algorithms.

However, existing simulators often suffer from a "Goldilocks" problem. Hand-designed simulators tend to be overly simplistic, lacking the messy, intricate nuances of real-world data. Conversely, purely data-driven simulators often lack "structural constraints"—the logical, underlying rules of cause and effect. This makes them poorly suited for testing whether an algorithm can correctly identify a specific causal path.

IncomeSCM aims to bridge this gap. By transforming the classic, static Adult income dataset into a dynamic, multi-year time-series simulator, it creates a "semi-synthetic" environment that is both data-rich and structurally sound. In doing so, it provides a challenging new benchmark that reveals a sobering truth about modern machine learning: models that excel at predicting outcomes are often surprisingly poor at estimating causal effects.

Methods

IncomeSCM employs a repeatable strategy for converting cross-sectional (snapshot) data into a Sequential Structural Causal Model (SCM). The generation process is built on two core principles:

  1. Fit real data where possible: The initial state of the system (e.g., a person's age, education, and income at the start of the simulation) is learned directly from the Adult income dataset using standard machine learning models like Random Forests and Logistic Regression.
  2. Compose simple mechanisms for complexity: Because the original snapshot data does not show how individuals change over time, "transition mechanisms" are hand-designed. For example, age deterministically increases by one each year, while the probability of finishing a degree depends probabilistically on current enrollment.
Overview of the IncomeSCM generation process, from base dataset to transition samplers.

By chaining these simple, year-to-year transitions over a 7-year horizon, the simulator generates complex, non-linear causal effects. Even though the individual "rules" are straightforward, their cumulative impact makes it exceedingly difficult for standard estimators to recover the true causal dynamics.

Datasets

  • Base Dataset: The well-known UCI Adult income dataset, containing approximately 30,000 samples of demographic and employment data from the 1994 Census.
  • New Benchmark (IncomeSCM-1.0): The resulting simulator and its synthetic dataset of 50,000 samples. It introduces a new variable, "Studies," which acts as the primary intervention (treatment) to evaluate how a year of full-time study impacts income six years later.

Key Insights and Ideas

  • The "Factual" Fallacy: One of the paper's most critical takeaways is that predictive accuracy does not equal causal accuracy. A model can be nearly perfect at predicting a person's actual, observed income (the factual outcome) but fail miserably at estimating the counterfactual effect of an intervention.
  • Structural Realism: By imposing a strict causal graph—a map defining exactly what causes what—the simulator allows researchers to rigorously test "identification strategies." This helps determine which variables must be adjusted for to successfully remove bias.
The initial-state causal graph ($G_1$) illustrating the structural complexity of the simulated environment.
  • Simplicity Leads to Complexity: The benchmark demonstrates that even when the "rules" of a system are known and relatively simple, their cumulative effect over time creates a functional complexity that challenges state-of-the-art causal estimators.

Key Results

The study evaluated several popular causal estimators on the IncomeSCM benchmark, including T-learners, S-learners, and Double Machine Learning (DML).

  • High Predictive Performance: Most models excelled at standard predictive tasks. They accurately predicted who would receive the "treatment" (AUC $\approx$ 0.98) and accurately forecasted observed outcomes ($R^2 \approx 0.80$).
  • Poor Causal Estimation: Despite stellar predictive scores, the models struggled significantly to estimate the Conditional Average Treatment Effect (CATE). The best-performing models only achieved an $R^2$ of about 0.30 to 0.33 for the causal effect.
  • No Universal Winner: No single method consistently outperformed the others across all tasks. A Random Forest T-learner might win on one specific task, while an XGBoost S-learner wins on another. This inconsistency highlights a pressing need for better model selection criteria in causal inference.

Limitations and Future Work

While IncomeSCM represents a significant step forward for causal benchmarking, it has limitations that pave the way for future research:

  • Censoring and Attrition: The current simulator assumes all subjects remain in the study for the full 7-year duration. Real-world data is plagued by people moving, dropping out, or passing away (censoring), which complicates causal analysis.
  • Missing Data: The simulator does not currently generate or account for missing values, another ubiquitous hurdle in applied research.
  • Hyperparameter Tuning: The models used to fit the initial state were not exhaustively optimized. More rigorous tuning could yield an even more realistic baseline distribution.

Future iterations of IncomeSCM could incorporate the effects of "missingness" and attrition. Furthermore, the benchmark serves as a call to action for the community to develop more robust model selection criteria, helping researchers know which causal estimator to trust when real-world ground truth remains unobservable.

Summary updated

Abstract

Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.
BibTeX
@inproceedings{johansson2024incomescm,
  author = {Fredrik D. Johansson},
  title = {{IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=GNhwwbZEZ7},
}
Figure