Abstract
Causal discovery aims to learn causal relationships between variables from targeted data, making it a fundamental task in machine learning. However, causal discovery algorithms often rely on unverifiable causal assumptions, which are usually difficult to satisfy in real-world data, thereby limiting the broad application of causal discovery in practical scenarios. Inspired by these considerations, this work extensively benchmarks the empirical performance of various mainstream causal discovery algorithms, which assume i.i.d. data, under eight model assumption violations. Our experimental results show that differentiable causal discovery methods exhibit robustness under the metrics of Structural Hamming Distance and Structural Intervention Distance of the inferred graphs in commonly used challenging scenarios, except for scale variation. We also provide the theoretical explanations for the performance of differentiable causal discovery methods. Finally, our work aims to comprehensively benchmark the performance of recent differentiable causal discovery methods under model assumption violations, and provide the standard for reasonable evaluation of causal discovery, as well as to further promote its application in real-world scenarios.
Introduction
Causal discovery is a cornerstone of scientific inquiry, seeking to uncover the directed acyclic graphs (DAGs) that govern complex systems. Traditionally, algorithms have relied on strict mathematical assumptions—such as causal sufficiency (no hidden confounders), independent and identically distributed (i.i.d.) data, and specific functional forms—to ensure identifiability. However, real-world data is rarely so well-behaved. While traditional constraint-based and score-based methods have been studied under misspecification, the robustness of the recent wave of differentiable causal discovery methods (e.g., NOTEARS, DAGMA) remains largely unexplored. This paper addresses this gap by benchmarking 12 mainstream algorithms across 8 common model assumption violations.
Methodology and Benchmarking
The authors conducted an extensive empirical study involving over 70,000 experiments. They evaluated 12 algorithms categorized into constraint-based (PC), score-based (GES), functional causal models (DirectLiNGAM, CAM, Var-SortnRegress, $R^2$-SortnRegress), and differentiable/gradient-based methods (NOTEARS, GOLEM, NOTEARS-MLP, GraN-DAG, NoCurl, DAGMA).
These methods were tested against eight specific misspecification scenarios:
- Latent Confounders: Unobserved variables affecting multiple observed nodes.
- Measurement Error: Added noise to observed values.
- Unfaithful Distribution: Causal paths that cancel each other out.
- Autoregressive Effects: Temporal dependencies violating i.i.d. assumptions.
- Heterogeneous Effects: Data from multiple domains with varying noise distributions.
- Scale-variant Effects: Standardized variables altering natural variance scales.
- Missing Mechanisms: Data missing completely at random (MCAR).
- Mechanism Violation: Mismatch between data generation (e.g., nonlinear) and model assumptions (e.g., linear).
Key Insights and Results
The Robustness of Differentiable Methods
A primary finding is that differentiable methods are surprisingly resilient. Despite being optimized for "vanilla" i.i.d. scenarios, gradient-based methods achieved optimal or highly competitive results in 7 out of 8 misspecified scenarios. They consistently outperformed traditional mainstays like PC and GES, particularly in the presence of latent confounders and measurement errors.
The Achilles Heel: Scale Variation
The most significant weakness identified for linear differentiable methods is scale variation. When variables are standardized or have vastly different scales, the performance of these methods drops drastically. The authors provide a theoretical explanation: scale variation alters the "noise ratio" between variables, which breaks the least-squares optimization guarantees that linear differentiable methods rely on to identify the true DAG.
Performance on Real-World Data
The benchmark extended to the biological Sachs dataset, where the differentiable method DAGMA achieved the best performance (SHD of 12, SID of 42), significantly outperforming PC (SHD 22) and GES (SHD 30). Furthermore, differentiable methods proved computationally efficient; for a 10-node graph, DAGMA required only ~2.4 seconds, compared to ~45 seconds for traditional nonlinear methods like CAM.
Conclusion
This work demonstrates that differentiable causal discovery methods are robust tools capable of handling many real-world data imperfections. While they excel in scenarios involving confounders and measurement errors, practitioners must remain cautious regarding variable scaling. Future research should focus on developing scale-invariant loss functions for nonlinear continuous optimization to further enhance the reliability of these methods in practical applications.
Summary updated