Semi-Synthetic Benchmarking for Causal Inference

115 concepts96 papers1 proposal

The Fundamental Dilemma of Causal Validation

In machine learning, evaluating a predictive model is straightforward: you withhold a test set, run your model on the inputs, and compare the predictions ($\hat{y}$) against the true labels ($y$). If your model predicts a patient's risk of heart disease, you can verify its accuracy using historical clinical outcomes.

Causal inference, however, breaks this paradigm. The goal of a causal model is not merely to predict an outcome, but to estimate the effect of an intervention—for instance, how much a specific medication changes a patient's blood pressure compared to a placebo. To know this with absolute certainty for an individual, we would need to observe two parallel universes: one where they took the drug, and one where they did not. This is known as the Fundamental Problem of Causal Inference. Because we only ever observe the factual outcome and never the counterfactual, we lack a true ground-truth label ($y$) for the causal effect.

Without ground truth, how can we validate causal estimators, tune hyperparameters, or compare different algorithmic architectures?

The Failure of Purely Synthetic Data

The simplest workaround is to simulate a dataset from scratch using basic mathematical equations (e.g., generating covariates from a standard Gaussian distribution and defining a linear relationship for the outcome). In this fully synthetic environment, the true causal effect is perfectly known because we wrote the equations ourselves.

However, this approach suffers from a severe reality gap. Real-world observational data is messy. It contains complex joint distributions, non-linearities, unobserved confounding, selection biases, and heavy-tailed noise. A causal estimator that performs flawlessly on a clean, simulated Gaussian dataset will often fail when applied to real electronic health records or economic indicators. Evaluating algorithms on oversimplified toy data creates a false sense of security and fails to reveal how models handle the nuances of empirical data.

The Semi-Synthetic Solution

To bridge this gap, researchers have developed semi-synthetic benchmarking. The core intuition is to preserve the rich, complex statistical properties of real-world covariates while overlaying a mathematically controlled causal mechanism. By anchoring the simulation to empirical data, the resulting benchmark tests algorithms against realistic data distributions while still providing an exact, verifiable ground truth.

There are two primary paradigms for constructing these semi-synthetic environments: Covariate-Preserving Outcome Modeling and Deep Generative Interventional Modeling.

Purely Synthetic Data         Semi-Synthetic Data          Real Observational Data
[Synthetic Covariates]     [Real-World Covariates]         [Real-World Covariates]
         +                            +                               +
[Synthetic Mechanism]      [Synthetic Mechanism]           [Unknown Real Mechanism]
         ||                           ||                              ||
(Easy to validate,         (Easy to validate,              (Impossible to validate,
 unrealistic distributions) realistic distributions)        realistic distributions)

1. Covariate-Preserving Outcome Modeling

In this paradigm, researchers harvest authentic features (covariates) directly from a real-world study. Because these features are untouched, they retain all the natural correlations, missingness, and anomalies of the target domain. The researchers then discard the original, unknown treatment assignment and outcome mechanisms, replacing them with expert-informed synthetic functions.

For example, Jonathan Crabbé et al. utilize real-world patient covariates paired with synthetic outcome functions to evaluate Heterogeneous Treatment Effect (HTE) models. Because the outcome function is explicitly defined by the researcher, they can isolate exactly which features are prognostic (predictive of the outcome regardless of treatment) versus predictive (determining the response to the treatment). Similarly, R. Teal Witter et al. combine authentic subject covariates with synthetic outcome functions derived from domain expertise to benchmark estimators for natural experiments.

By controlling the functional form of the outcome, researchers can introduce specific challenges—such as non-linear treatment responses or high-dimensional interactions—to systematically stress-test where an estimator breaks down.

2. Deep Generative Interventional Modeling

While preserving covariates maintains real-world feature distributions, manually designing the outcome and treatment equations can still introduce artificial simplicity. The second paradigm solves this by training deep generative models directly on the full observational dataset—covariates, treatments, and outcomes combined.

Once a generative model accurately captures the joint distribution of the real data, it can be used as a proxy for the real world. Crucially, because the generative model is fully parameterized, researchers can mathematically derive or sample from its interventional distributions, effectively unlocking the counterfactuals.

Several architectures have been adapted to achieve this:

  • Conditional VAEs: The "Credence" framework introduced by Harsh Parikh et al. uses Conditional Variational Autoencoders to generate synthetic datasets that are stochastically indistinguishable from a researcher's empirical sample. By embedding user-defined causal functions directly into the generative loss function, it ensures the output adheres to a known ground truth while retaining the exact non-linearities and outliers of the original study.
  • Normalizing Flows: To avoid the soft constraints of VAEs, Daniel de Vassimon Manela et al. developed "Frugal Flows," which combine normalizing flows with copulas. This architecture allows researchers to exactly parameterize marginal causal effects (like the Average Treatment Effect) directly into the probability distribution, ensuring the synthetic data satisfies the target causal truth without requiring post-hoc validation.
  • Flexible Neural Samplers: The "RealCause" benchmark, developed by Brady Neal et al., fits highly flexible neural networks to real-world observational data to learn the relationships between covariates, treatments, and outcomes. The authors introduce tunable knobs into the generation process, allowing users to systematically alter characteristics like positivity/overlap violations and treatment heterogeneity to evaluate model robustness.

Domain-Specific Structural Simulations

In fields governed by physical laws or structural constraints—such as climate science, astrophysics, or neuroscience—purely statistical generative models might violate fundamental domain principles. In these settings, semi-synthetic benchmarking relies on domain-specific mathematical simulators.

In climate science, for instance, Xavier‐Andoni Tibau et al. introduced a spatiotemporal stochastic climate model that simulates low-dimensional latent variables interacting through predefined causal equations before projecting them onto a high-dimensional spatial grid. This allows for the rigorous benchmarking of causal discovery algorithms on data that behaves like real atmospheric teleconnections.

Similarly, in astrophysics, Zehao Jin et al. validate causal discovery methods using semi-analytic models where synthetic datasets are generated from cosmic simulations with hard-coded causal directions. As argued by Clark Glymour et al., these domain-specific simulations serve as a vital proxy for experimental ground truth in scientific fields where physical experimentation is impossible.

Why It Works: Disentangling Realism from Verification

The power of semi-synthetic benchmarking lies in its ability to separate the statistical complexity of data from the unverifiability of the causal mechanism.

By leveraging techniques like separating causal dependencies from unexplained residual terms—as demonstrated in time-series data by Yuxiao Cheng et al.—semi-synthetic frameworks prove that data realism and ground-truth verifiability are not mutually exclusive. They allow practitioners to construct an algorithmic "unit test" or sandbox. Within this sandbox, we can confidently rank estimators, diagnose when a model is fooled by a spurious correlation, and ensure that when a model claims a causal relationship exists, it is reflecting structural truth rather than statistical noise.

Key Foundational Work

Automating Path Analysis for Building Causal Models from Data (1993) — contributed 1 concept elements to this direction.
Causality: Objectives and Assessment (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semi_synthetic_causal_benchmarking_33ce385c,
  title = {semi-synthetic causal benchmarking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semi-synthetic-causal-benchmarking/rdr_33ce385cac1b431e8113c8b9f7097f7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semi_synthetic_causal_benchmarking_33ce385c_v1,
  title = {semi-synthetic causal benchmarking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semi-synthetic-causal-benchmarking/rdr_33ce385cac1b431e8113c8b9f7097f7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake