Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation

Ruibo Tu, Kun Zhang, Bo C. Bertilson, Hedvig Kjellström, Cheng Zhang

Causal discovery—the process of identifying cause-and-effect relationships from observational data—is a holy grail for fields ranging from medicine to climate science. However, the field faces a major "ground truth" problem: in the real world, we rarely know the true causal structure, making it nearly impossible to verify if our algorithms actually work.

In a NeurIPS 2019 paper, researchers from KTH, CMU, and Microsoft Research introduce a novel solution: a high-fidelity Neuropathic Pain Diagnosis Simulator. By leveraging well-understood medical pathophysiology, they created a benchmark that bridges the gap between simplistic "toy" simulations and unverifiable real-world datasets.

The Ground Truth Dilemma

In supervised learning, we have clear labels (e.g., "this image is a cat"). In causal discovery, we need "causal labels" (e.g., "A causes B"), which are notoriously difficult to obtain. Randomized controlled trials—the gold standard for establishing causality—are often unethical, expensive, or impossible to conduct.

Existing benchmarks usually fall into two problematic camps:

  1. Purely Synthetic: Randomly generated graphs that don't reflect the "physics" or logic of real-world systems.
  2. Real-World Data: Datasets where the "true" causal graph is just an educated guess by experts, making evaluation circular.
Dermatome map illustrating the biological wiring of nerve roots to specific skin areas, serving as the basis for the causal graph.

The authors tackle this by using neuropathic pain—a condition where the biological "wiring" (how nerve injuries lead to specific symptoms) is exceptionally well-mapped in medical literature. Medicine acts as a natural causal graph; pathophysiology is essentially a massive, pre-validated map of cause and effect.

Building the Simulator

The researchers built the simulator using a three-layered causal structure to generate synthetic patient data based on a ground-truth causal graph of 222 variables and approximately 800 relations.

The three-layered causal structure of the simulator: Pathophysiology (root causes), Pattern (nerve sets), and Symptom (resulting discomfort).

The three layers include:

  1. Pathophysiology: The root causes, such as discoligamentous injuries (DLI) in the spine.
  2. Pattern (Radiculopathy): How those injuries affect specific nerve sets.
  3. Symptoms: The resulting localized discomfort (e.g., "left neck pain" or "left lateral arm pain").

Key Technical Innovations

  • Parameter Estimation from Small Data: They used a real-world dataset of 141 clinical records from a specialized hospital department to estimate the parameters of the graph.
  • Heuristic CPD Estimation: To avoid the "curse of dimensionality" (where variables have too many parents to estimate probabilities accurately from a small sample), they introduced a heuristic: the probability of a symptom occurring is approximated by the maximum conditional probability given any single active cause.
  • Ancestral Sampling: Once the graph and probabilities were set, they used ancestral sampling to generate infinite "synthetic patients."
  • Stress Testing: The simulator includes modules to intentionally introduce unmeasured confounders (e.g., hiding a parent node), selection bias (e.g., only "hospitalizing" certain patients), and missing data mechanisms (MCAR, MAR, and MNAR).

Key Results and Insights

To validate the simulator, the authors conducted a "Turing Test" for medical data. A specialized physician was asked to distinguish between 50 real and 50 synthetic records. Not only could the physician not tell the difference, but they actually rated the synthetic records as more "typical" than the real-world records. This "Typicality Paradox" suggests the simulator successfully captured the core logic of the disease while smoothing out the idiosyncratic noise found in tiny real-world samples.

The authors then used the simulator to benchmark four major causal discovery algorithms (PC, FCI, RFCI, and GES):

  • Sample Size Sensitivity: Performance (F1 score and Causal Accuracy) improved with more data but remained surprisingly low even at 16,000+ samples. This highlights the sheer difficulty of causal discovery in high-dimensional spaces (200+ variables).
  • Algorithm Comparisons: The Greedy Equivalence Search (GES) generally outperformed the PC algorithm in causal accuracy across most settings. Meanwhile, RFCI proved more robust to selection bias than the standard FCI algorithm, aligning with its theoretical design.
  • Missing Data: Standard algorithms struggled significantly when data was "Missing Not At Random" (MNAR), a common occurrence in healthcare datasets.

Limitations and Future Work

While a massive step forward, the simulator has limitations. The "max-probability" heuristic for conditional distributions is a simplification of how multiple causes interact biologically. Furthermore, the current model focuses primarily on discoligamentous injuries, excluding rarer causes like tumors or diabetes.

Future work will likely expand the simulator to include these "edge case" scenarios and apply similar methodologies to other medical domains where causal links are well-documented, providing the machine learning community with even more robust tools for evaluating causal discovery.

Paper Citation: Tu, R., Zhang, K., Bertilson, B. C., Kjellström, H., & Zhang, C. (2019). Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).

Summary updated

Abstract

Discovery of causal relations from observational data is essential for many disciplines of science and real-world applications. However, unlike other machine learning algorithms, whose development has been greatly fostered by a large amount of available benchmark datasets, causal discovery algorithms are notoriously difficult to be systematically evaluated because few datasets with known ground-truth causal relations are available. In this work, we handle the problem of evaluating causal discovery algorithms by building a flexible simulator in the medical setting. We develop a neuropathic pain diagnosis simulator, inspired by the fact that the biological processes of neuropathic pathophysiology are well studied with well-understood causal influences. Our simulator exploits the causal graph of the neuropathic pain pathology and its parameters in the generator are estimated from real-life patient cases. We show that the data generated from our simulator have similar statistics as real-world data. As a clear advantage, the simulator can produce infinite samples without jeopardizing the privacy of real-world patients. Our simulator provides a natural tool for evaluating various types of causal discovery algorithms, including those to deal with practical issues in causal discovery, such as unknown confounders, selection bias, and missing data. Using our simulator, we have evaluated extensively causal discovery algorithms under various settings.
BibTeX
@inproceedings{tu2019neuropathic,
  author = {Ruibo Tu and Kun Zhang and Bo C. Bertilson and Hedvig Kjellström and Cheng Zhang},
  title = {{Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure