Introduction
In the world of causal inference, establishing "ground truth" is the ultimate challenge. To determine if a specific intervention—like a job training program—actually works, researchers must estimate the "counterfactual": what would have happened to the exact same person had they not participated. Because we cannot observe two realities simultaneously, the field relies heavily on simulators to test and validate causal algorithms.
However, existing simulators often suffer from a "Goldilocks" problem. Hand-designed simulators tend to be overly simplistic, lacking the messy, intricate nuances of real-world data. Conversely, purely data-driven simulators often lack "structural constraints"—the logical, underlying rules of cause and effect. This makes them poorly suited for testing whether an algorithm can correctly identify a specific causal path.
IncomeSCM aims to bridge this gap. By transforming the classic, static Adult income dataset into a dynamic, multi-year time-series simulator, it creates a "semi-synthetic" environment that is both data-rich and structurally sound. In doing so, it provides a challenging new benchmark that reveals a sobering truth about modern machine learning: models that excel at predicting outcomes are often surprisingly poor at estimating causal effects.
Methods
IncomeSCM employs a repeatable strategy for converting cross-sectional (snapshot) data into a Sequential Structural Causal Model (SCM). The generation process is built on two core principles:
- Fit real data where possible: The initial state of the system (e.g., a person's age, education, and income at the start of the simulation) is learned directly from the Adult income dataset using standard machine learning models like Random Forests and Logistic Regression.
- Compose simple mechanisms for complexity: Because the original snapshot data does not show how individuals change over time, "transition mechanisms" are hand-designed. For example, age deterministically increases by one each year, while the probability of finishing a degree depends probabilistically on current enrollment.
By chaining these simple, year-to-year transitions over a 7-year horizon, the simulator generates complex, non-linear causal effects. Even though the individual "rules" are straightforward, their cumulative impact makes it exceedingly difficult for standard estimators to recover the true causal dynamics.
Datasets
- Base Dataset: The well-known UCI Adult income dataset, containing approximately 30,000 samples of demographic and employment data from the 1994 Census.
- New Benchmark (IncomeSCM-1.0): The resulting simulator and its synthetic dataset of 50,000 samples. It introduces a new variable, "Studies," which acts as the primary intervention (treatment) to evaluate how a year of full-time study impacts income six years later.
Key Insights and Ideas
- The "Factual" Fallacy: One of the paper's most critical takeaways is that predictive accuracy does not equal causal accuracy. A model can be nearly perfect at predicting a person's actual, observed income (the factual outcome) but fail miserably at estimating the counterfactual effect of an intervention.
- Structural Realism: By imposing a strict causal graph—a map defining exactly what causes what—the simulator allows researchers to rigorously test "identification strategies." This helps determine which variables must be adjusted for to successfully remove bias.
- Simplicity Leads to Complexity: The benchmark demonstrates that even when the "rules" of a system are known and relatively simple, their cumulative effect over time creates a functional complexity that challenges state-of-the-art causal estimators.
Key Results
The study evaluated several popular causal estimators on the IncomeSCM benchmark, including T-learners, S-learners, and Double Machine Learning (DML).
- High Predictive Performance: Most models excelled at standard predictive tasks. They accurately predicted who would receive the "treatment" (AUC $\approx$ 0.98) and accurately forecasted observed outcomes ($R^2 \approx 0.80$).
- Poor Causal Estimation: Despite stellar predictive scores, the models struggled significantly to estimate the Conditional Average Treatment Effect (CATE). The best-performing models only achieved an $R^2$ of about 0.30 to 0.33 for the causal effect.
- No Universal Winner: No single method consistently outperformed the others across all tasks. A Random Forest T-learner might win on one specific task, while an XGBoost S-learner wins on another. This inconsistency highlights a pressing need for better model selection criteria in causal inference.
Limitations and Future Work
While IncomeSCM represents a significant step forward for causal benchmarking, it has limitations that pave the way for future research:
- Censoring and Attrition: The current simulator assumes all subjects remain in the study for the full 7-year duration. Real-world data is plagued by people moving, dropping out, or passing away (censoring), which complicates causal analysis.
- Missing Data: The simulator does not currently generate or account for missing values, another ubiquitous hurdle in applied research.
- Hyperparameter Tuning: The models used to fit the initial state were not exhaustively optimized. More rigorous tuning could yield an even more realistic baseline distribution.
Future iterations of IncomeSCM could incorporate the effects of "missingness" and attrition. Furthermore, the benchmark serves as a call to action for the community to develop more robust model selection criteria, helping researchers know which causal estimator to trust when real-world ground truth remains unobservable.
Summary updated