Abstract
Discovery of causal relations from observational data is essential for many disciplines of science and real-world applications. However, unlike other machine learning algorithms, whose development has been greatly fostered by a large amount of available benchmark datasets, causal discovery algorithms are notoriously difficult to be systematically evaluated because few datasets with known ground-truth causal relations are available. In this work, we handle the problem of evaluating causal discovery algorithms by building a flexible simulator in the medical setting. We develop a neuropathic pain diagnosis simulator, inspired by the fact that the biological processes of neuropathic pathophysiology are well studied with well-understood causal influences. Our simulator exploits the causal graph of the neuropathic pain pathology and its parameters in the generator are estimated from real-life patient cases. We show that the data generated from our simulator have similar statistics as real-world data. As a clear advantage, the simulator can produce infinite samples without jeopardizing the privacy of real-world patients. Our simulator provides a natural tool for evaluating various types of causal discovery algorithms, including those to deal with practical issues in causal discovery, such as unknown confounders, selection bias, and missing data. Using our simulator, we have evaluated extensively causal discovery algorithms under various settings.
Introduction
In standard machine learning, massive benchmarks like ImageNet provide clear performance metrics. Causal discovery—the field of identifying cause-and-effect relationships from purely observational data—lacks this luxury. Verifying if an algorithm correctly identifies that "A causes B" typically requires experimental interventions or randomized controlled trials, which are often unethical or impossible in fields like medicine.
Existing benchmarks for causal discovery are often either too small, purely synthetic (and thus unrealistic), or lack verified ground truth. This gap makes it difficult to identify which algorithms perform effectively in complex, "messy" real-world scenarios. This research addresses this by turning to a domain where causal mechanisms are biologically mapped: neuropathic pain.
Methodology: The Neuropathic Pain Diagnosis Simulator
The authors constructed a Neuropathic Pain Diagnosis Simulator by integrating expert medical knowledge with real-world clinical data. The core of the simulator is a three-layer Directed Acyclic Graph (DAG) representing the causal hierarchy of the condition:
- Pathophysiology: Root causes, such as disc injuries (e.g., DLI C4-C5).
- Pattern (Radiculopathy): Specific nerves affected by the pathophysiology (e.g., C5 nerve root).
- Symptom: The resulting discomfort felt by the patient in specific anatomical regions (e.g., neck pain, lateral arm numbness).
To ensure realism, Conditional Probability Distributions (CPDs) were estimated from a dataset of 141 real clinical records. For nodes with many parents where data was sparse, a "max" heuristic was implemented, assuming the probability of a symptom is at least as high as the probability from its most likely single cause—a logic aligned with medical intuition.
The simulator also allows for the controlled introduction of practical data challenges, such as unmeasured confounding (hiding parent nodes), selection bias (filtering for specific patient populations), and various missing data mechanisms.
Validation and Results
The simulator's utility depends on its ability to generate data that mirrors real-world complexity. Validation was performed through statistical comparison and expert review.
Data Fidelity
The marginal distributions and co-occurrence matrices of the simulated data closely matched those of the real-world clinical dataset. This statistical alignment confirms the simulator's effectiveness as a proxy for real medical data while bypassing privacy concerns.
Furthermore, in a "Turing Test" for data, a specialist physician was unable to distinguish between 50 real and 50 simulated records, even rating the simulated records as slightly more "typical" due to the absence of noise and missing labels common in messy clinical notes.
Algorithm Benchmarking
The authors evaluated several major causal discovery algorithms, including PC, FCI, GES, and RFCI. Key findings include:
- Sample Efficiency: Even with over 16,000 samples, standard algorithms struggled to achieve high causal accuracy, highlighting a significant need for more sample-efficient methods.
- Robustness: GES (score-based) generally outperformed PC (constraint-based) under selection bias. RFCI proved more robust to selection bias than FCI, despite both being designed for latent variable scenarios.
Conclusion
By leveraging the well-understood causal links in human anatomy and neuropathic pathophysiology, this simulator provides a rigorous benchmark for causal discovery. It generates realistic, high-dimensional data (222 nodes, 770 edges) that allows researchers to test algorithms against unmeasured confounding, selection bias, and missing data. Future work will focus on extending the model to include temporal dynamics and feedback loops, further bridging the gap between synthetic benchmarks and the complexities of chronic medical conditions.
Summary updated