The Synthetic to Real Gap in Causal Discovery

35 concepts28 papers1 proposal

Causal discovery aims to uncover the true structural mechanisms generating data, moving beyond mere statistical correlation to understand cause-and-effect relationships. Historically, this relied on rigid constraint-based or score-based statistical tests. Recently, however, the field has shifted toward framing causal discovery as a machine learning problem—utilizing supervised learning, meta-learning, and deep representation learning to predict causal graphs directly from observational data.

While these modern machine learning approaches achieve near-perfect accuracy on standard synthetic benchmarks, they frequently collapse when deployed on real-world data. This discrepancy is known as the synthetic-to-real generalization gap (or the causal reality gap). Understanding why this gap exists and how to bridge it is one of the most critical challenges in making causal AI practical.


Why Machine Learning Approaches to Causality Overfit

To understand why the reality gap occurs, we must look at how modern causal discovery models are trained.

David Lopez-Paz et al. in Towards a Learning Theory of Cause-Effect Inference and David López-Paz et al. in The Randomized Causation Coefficient pioneered the idea of treating cause-effect inference as a supervised learning problem. By transforming probability distributions into low-dimensional feature vectors using kernel mean embeddings, a classifier can be trained to look at a dataset of variables $(X, Y)$ and output a label: $X \to Y$ or $Y \to X$.

However, training a supervised model or a foundation model requires thousands of examples with known ground-truth causal graphs. Because ground-truth causal labels are exceptionally rare in the real world, researchers rely almost entirely on synthetic data generated by explicit Structural Equation Models (SEMs) to train these networks.

This reliance creates a fundamental vulnerability. As Yujia Bao et al. observe in Sample, estimate, aggregate: A recipe for causal discovery foundation models, when the underlying functional mechanisms of real-world systems differ from the synthetic distributions used during training, the simulation-to-reality gap severely limits the model's utility.

The same issue plagues meta-learning frameworks. For instance, the Transformer Neural Process framework developed by Anish Dhir et al. in Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learning learns to predict interventional distributions from observational data by training on synthetic causal priors. If the target domain's actual graph structures or functional dynamics fall outside this synthetic prior, the model's predictive accuracy degrades sharply. The model overfits to the style of the synthetic data generator rather than learning universal laws of causality.


The Anatomy of Real-World Complexity

Why exactly do synthetic data generators fail to prepare models for the real world? Synthetic benchmarks typically generate data using clean, idealized mathematical laws—such as linear or polynomial equations with additive, independent Gaussian noise. Real-world systems violate these assumptions in several distinct ways:

1. Non-Linearity and Non-Stationarity

Real-world systems evolve over time and exhibit highly complex, non-linear interactions. Yuxiao Cheng et al. demonstrate in CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery that time-series algorithms achieving near-perfect accuracy on simplified synthetic benchmarks experience severe performance drops on realistic data distributions. The models become over-optimized to the stable, predictable mathematical simulations of toy datasets and fail to generalize to the messy, non-stationary dynamics of actual observations.

2. Unmodeled Noise and Complex Agent Behaviors

In a simulation, noise is a controlled, independent variable. In reality, noise is often structured, environmental, or caused by unmeasured variables. When evaluating causal discovery on road driver behavior, Rhys Peter Matthew Howard et al. in Evaluating Temporal Observation-Based Causal Discovery Techniques Applied to Road Driver Behaviour found that algorithms optimized for idealized conditions fail when confronted with the unmodeled noise, measurement errors, and complex goal-directed behaviors of human agents.

3. The Breakdown of Causal Representation Learning

Causal Representation Learning (CRL) attempts to solve an even harder problem: discovering latent causal variables directly from raw, unstructured data (like pixels). To see if these methods hold up outside of simulations, Juan L. Gamella et al. conducted a sanity check in Sanity Checking Causal Representation Learning on a Simple Real-World System using a basic physical optical system. They discovered that state-of-the-art CRL methods failed catastrophically to recover the known physical ground-truth factors, demonstrating that models performing well under idealized mathematical assumptions are rarely robust enough for physical reality.


Exposing the Gap via Empirical Benchmarks

To quantify this reality gap, researchers have begun shifting away from synthetic data generation entirely, building large-scale benchmarks grounded in physical, biological, or experimental ground truths. These benchmarks consistently expose the limitations of current theories:


Pathways to Bridging the Gap

To build causal discovery models that successfully generalize to real-world applications, researchers are developing frameworks that ground validation and training in empirical data rather than handcrafted equations.

Grounding in Experimental Data

One approach is to repurpose real-world experimental data to validate observational methods. Amanda Gentzel et al. introduced the OSRCT framework in How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inference. By taking data from actual Randomized Controlled Trials (RCTs) and systematically inducing artificial confounding, the authors created realistic observational benchmarks. Interestingly, they found that while synthetic datasets show massive performance gaps between various causal algorithms, these differences largely diminish when evaluated on data derived from actual physical experiments.

Empirical Generative Modeling

Rather than guessing the mathematical equations that govern a real-world system, practitioners can use deep generative models to mimic the target domain's distribution. Harsh Parikh et al. developed the "Credence" framework in Validating Causal Inference Methods, which uses Conditional Variational Autoencoders (CVAEs) to generate synthetic datasets that are statistically indistinguishable from a researcher's real-world empirical data, but contain known, user-defined causal truths.

[Real-World Observational Data] ──> [Train Conditional VAE] ──> [Empirical Synthetic Data]
                                                                          │
                                                                  (Inject Known Causal Truths)
                                                                          │
                                                                          ▼
                                                              [Stress-Test Causal Model]

By stress-testing causal discovery algorithms on these custom, data-specific simulations before applying them to the actual observed sample, practitioners can bypass the brittle assumptions of traditional synthetic data and ensure their models are robust enough for the messiness of reality.

Key Foundational Work

The Randomized Causation Coefficient (2014) — contributed 2 concept elements to this direction.
Towards a Learning Theory of Cause-Effect Inference (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_synthetic_to_real_generalization_gap_syn_6e66ca89,
  title = {synthetic-to-real generalization gap (synthetic-to-real gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-to-real-generalization-gap-synthetic-to-real-gap/rdr_6e66ca8966444f36833d05ecd4307906},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_synthetic_to_real_generalization_gap_syn_6e66ca89_v1,
  title = {synthetic-to-real generalization gap (synthetic-to-real gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-to-real-generalization-gap-synthetic-to-real-gap/rdr_6e66ca8966444f36833d05ecd4307906/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake