Imagine two competing blueprints of a city's water infrastructure. Blueprint A is nearly perfect but forgets a single pipe leading to an isolated suburban house. Blueprint B is also nearly perfect but accidentally reverses the direction of the main check valve, implying that water flows backward from the city into the reservoir.
If you evaluate these blueprints purely by counting structural errors—a metric analogous to the Structural Hamming Distance (SHD) commonly used in causal discovery—both blueprints look equally flawed: each contains exactly one error. However, if you attempt to use these blueprints to plan an intervention, such as shutting down a valve for maintenance, Blueprint A remains 99% functional, while Blueprint B leads to a catastrophic systemic failure.
This disconnect highlights the core limitation of traditional causal graph evaluation. For years, the field of causal discovery has focused on finding the "true" Directed Acyclic Graph (DAG), measuring success by how many edges an algorithm correctly identifies or orientates. Yet, as Amanda Gentzel et al. demonstrate in The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data, topological precision does not inherently guarantee functional utility. A model can excel at edge-detection metrics while failing completely to predict the quantitative impact of an intervention.
To bridge this gap, a growing body of research is shifting the paradigm away from purely structural metrics and toward functional and interventional evaluation frameworks.
Why Structural Metrics Mislead
Counting edge differences fails to capture the true distance between causal models because edges are not logically or functionally uniform. A single structural change can have wildly asymmetric consequences depending on where it occurs and what role it plays in downstream inference.
The Logical Disconnect
Structural metrics treat every edge independently, ignoring how they combine to form conditional independencies. In Separation-Based Distance Measures for Causal Graphs, Jonas Wahl et al. point out that reversing a single edge can fundamentally alter the entire logical meaning (the d-separation statements) of a graph. By evaluating graphs based on conditional independence separations rather than raw edge counts, we can penalize structural errors according to how severely they distort the graph's underlying causal logic.
The Causal Formula Disconnect
When a causal graph is used to estimate a treatment effect, it dictates an identification strategy, such as the back-door adjustment criterion. Sebastian Weichwald et al. exploit this insight in Adjustment Identification Distance: A gadjid for Causal Structure Learning. They argue that if a missing or reversed edge does not alter the validity of the downstream adjustment set, the error is functionally benign. Conversely, an error that introduces an unblocked confounding path completely invalidates the resulting causal formula. Functional metrics must therefore evaluate a learned graph by verifying the correctness of the downstream causal formulas it generates.
True Graph: X ----> Y ----> Z (Adjustment set for X->Z is {Y} or empty)
Learned Graph A: X ----> Y Z (Missing edge: completely breaks X->Z effect formula)
Learned Graph B: X ----> Y <---- Z (Reversed edge: changes collider structure, invalidates adjustment)
Why Observational Predictive Metrics Fail
If structural metrics are fragile, a natural alternative might be to use standard machine learning validation techniques, such as cross-validation or likelihood maximization on held-out data. However, observational predictive accuracy is an unreliable proxy for causal validity.
As Jussi Viinikka et al. show in Intersection-Validation: A Method for Evaluating Structure Learning without Ground Truth, algorithms can achieve exceptional precision at estimating an observational probability distribution while failing entirely to identify the correct underlying causal relationships. This occurs because multiple distinct causal graphs can belong to the same Markov equivalence class, meaning they generate identical observational distributions.
When an external action manipulates the system, it induces an out-of-distribution shift. The authors of Protocol for the development of a reporting guideline for causal and counterfactual prediction models in biomedicine emphasize that standard predictive metrics break down under these conditions because they cannot account for the structural changes that occur when a treatment regime is actively modified.
The Interventional Paradigm
To truly evaluate a causal model, we must measure its performance on the very task it was designed for: predicting the behavior of a system under intervention. This has led to the development of several distinct, interventional distance metrics.
Downstream Utility and Treatment Effects
Instead of evaluating the graph as an end product, we can evaluate it based on its utility in decision-making tasks. Chris Chinenye Emezue et al. formalize this in Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation, quantifying graph errors by their direct impact on the accuracy of Average Treatment Effect (ATE) estimations. This approach ensures that the evaluation metric directly aligns with the practical goals of the end-user.
Causal Risk and Descendant Sets
Rather than verifying individual direct edges, we can evaluate a model's ability to capture the broader, systemic flow of causation. Marco Eigenmann et al. introduce a decision-theoretic framework in Evaluation of Causal Structure Learning Algorithms via Risk Estimation that utilizes the descendant-set Jaccard distance. Instead of checking if $X$ causes $Y$ directly, this metric calculates the similarity between the set of nodes a model predicts will be downstream descendants of an intervention and the set of nodes that are empirically affected. This prioritizes macro-level reachability over micro-level edge placement, providing a more stable and robust assessment of causal flow.
$$\text{Jaccard Distance} = 1 - \frac{|\text{Predicted Descendants} \cap \text{Empirical Descendants}|}{|\text{Predicted Descendants} \cup \text{Empirical Descendants}|}$$
Interventional Distribution Distances
When empirical interventional data is available, we can directly measure the distance between the predicted and observed post-intervention distributions. Jonas Bernhard Wildberger et al. propose the Interventional Kullback-Leibler (IKL) divergence in On the Interventional Kullback-Leibler Divergence. By aggregating data across multiple distinct interventional environments, the IKL divergence simultaneously captures both structural correctness and the precise magnitude of causal effects.
In high-dimensional biological applications where a complete ground-truth graph is impossible to obtain, this approach becomes essential. For instance, in CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data, Mathieu Chevalley et al. use the Wasserstein distance to measure the statistical shifts in gene expression profiles following real-world CRISPR perturbations. This allows researchers to rank causal discovery algorithms based on their empirical predictive performance rather than simulated topological metrics.
Moving Beyond the True DAG
Evaluating models by their interventional fidelity also frees us from the rigid assumption that nature must be represented as a simple, sparse DAG. As David Duvenaud et al. argued early on in Causal learning without DAGs, real-world systems are often too messy, continuous, or feedback-driven for traditional graph structures.
By shifting our evaluation metrics from structural identity (matching a target graph) to functional alignment (matching an interventional distribution), we can objectively evaluate and utilize non-DAG models, mixture models, and deep latent variable architectures. Ultimately, a causal model is only as good as its ability to answer the fundamental question of causation: "What happens if we change the system?"
Key Foundational Work
Design and Analysis of the Causation and Prediction Challenge (2008) — contributed 1 concept elements to this direction.
Causal learning without DAGs (2010) — contributed 1 concept elements to this direction.