Validating Causal Discovery Without Ground Truth

55 concepts52 papers1 proposal

In standard machine learning, evaluating a model is straightforward: you split your data into a training set and a holdout test set. If the model accurately predicts the labels on the test set, it generalizes well. This supervised paradigm works because the ground truth labels exist and are observable.

Causal discovery—the task of inferring the underlying directed graph of cause-and-effect relationships from data—enjoys no such luxury. In real-world observational data, the true causal graph is almost always an unobserved, latent property of nature. As noted by Erich Kummerfeld et al. in a summary of the CAWS round table discussion, causal discovery is fundamentally unsupervised; we cannot rely on standard holdout validation because the true graph is rarely observable.

Why can't we just use predictive accuracy as a proxy for causal truth? Consider a stable confounder that correlates two variables. A predictive model can exploit this correlation to achieve perfect test accuracy. However, if we intervene on the system, that correlation collapses. A model that captures mere correlation will fail under intervention, yet without a known ground truth, we cannot easily verify whether an algorithm has mapped a true causal mechanism or just a highly sophisticated correlation.

To bridge this verification gap, researchers have developed clever strategies to validate causal models without relying on an explicit ground-truth map.

Turning Causal Graphs into Predictive Hypotheses

One powerful intuition is to translate structural properties of a causal graph into testable statistical predictions. A causal graph is not just an abstract diagram; it implies a strict set of conditional independence relationships among variables. Most notably, a node's Markov boundary (or Markov blanket) consists of its parents, its children, and its children's other parents. Given its Markov boundary, a variable becomes conditionally independent of all other variables in the network.

This insight forms the foundation of Out-of-Sample Causal Tuning (OCT), an evaluation framework introduced by Konstantina Biza et al. in Out-of-Sample Tuning for Causal Discovery and Tuning Causal Discovery Algorithms.

Instead of checking if a learned graph matches an unknown ground truth, OCT treats the graph as a collection of localized predictive models. It isolates the Markov boundaries induced by the learned graph and evaluates their out-of-sample predictive performance on holdout data. If an algorithm infers an incorrect causal structure, its implied Markov boundaries will either omit true causes or include irrelevant variables, directly degrading its out-of-sample predictive power. This elegantly transforms an unsupervised structural validation problem into a supervised predictive one.

[True Graph (Unknown)] ----> Implies ----> [Perfect Markov Boundaries]
                                                    |
[Learned Graph]        ----> Implies ----> [Flawed Markov Boundaries] 
                                                    |
                                                    v
                                      (Evaluated via Out-of-Sample 
                                       Predictive Performance)

Testing Internal Consistency via Self-Compatibility

What if we want to evaluate a causal discovery algorithm purely on its structural merits without relying on downstream regression models? Another approach exploits the mathematical constraints of the causal discovery process itself through the lens of invariance.

In Self-compatibility: Evaluating causal discovery without ground truth, Philipp Michael Faller et al. introduce a framework that falsifies incorrect causal models by checking their consistency across different subsets of variables. The underlying intuition is simple: if an algorithm is truly uncovering a stable physical mechanism, its structural output should not radically contradict itself when it views different slices of the same system.

If you run a causal discovery algorithm on a full set of variables, and then run it again on a subset of those variables, the relationships among the remaining variables should remain compatible with the original graph. If the algorithm yields wildly conflicting graphs across these overlapping subsets, it is internally inconsistent. This "self-compatibility" check allows researchers to falsify unreliable algorithms entirely from observational data, without ever needing to see the "right" answer.

Anchoring to Physical Laws and Generative Flows

When statistical proxies are insufficient, the final line of defense against the verification gap is to anchor evaluation in domains where the ground truth is either physically guaranteed or mathematically controlled.

One approach is to leverage real-world systems governed by immutable physical laws. For example, Gideon Stein et al. developed CausalRivers - Scaling up benchmarking of causal discovery for real-world time-series, a benchmark that uses the geography and fluid dynamics of river networks as an objective, real-world ground truth. Because water physically flows downstream, the directional causal links between measurement stations are indisputable. Similarly, Jonas Peters et al. built physical "causal chambers"—controlled wind and light tunnels—to generate real-world datasets with verifiable physical mechanisms.

Alternatively, when real physical systems cannot be instrumented, researchers use advanced generative modeling to bridge the gap between oversimplified synthetic data and messy real-world distributions. In Marginal Causal Flows for Validation and Inference, the authors combine normalizing flows with copulas to learn complex, high-dimensional real-world data distributions while exactly parameterizing marginal causal effects. This allows them to generate high-fidelity synthetic benchmarks where the true Average Treatment Effect (ATE) is perfectly known, yet the data retains the non-linear complexities of the real world. In a similar vein, Yuxiao Cheng developed CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery to provide realistic time-series data with verifiable underlying structures.

By combining predictive proxies, internal consistency checks, and physics-bounded benchmarks, the causal inference community is turning an inherently unverifiable problem into a rigorously testable science.

Key Foundational Work

A Simulation Study of Three Related Causal Data Mining Algorithms (2001) — contributed 1 concept elements to this direction.
Generalized Do-Calculus with Testable Causal Assumptions (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_ground_truth_scarcity_42bbf604,
  title = {causal ground truth scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-ground-truth-scarcity/rdr_42bbf6046340451fba6562125d8befd6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_ground_truth_scarcity_42bbf604_v1,
  title = {causal ground truth scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-ground-truth-scarcity/rdr_42bbf6046340451fba6562125d8befd6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake