ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects

Newton Mwai Kinyanjui, Fredrik D. JohanssonarXiv 2021 · OpenReview

TL;DR

This paper introduces ADCB, a novel simulator for Alzheimer's disease that bridges the gap between overly simplistic causal inference benchmarks and the complexity of real-world healthcare data. By fitting to the ADNI dataset and incorporating realistic treatment effects, ADCB provides a more challenging and relevant platform for evaluating causal effect estimators.

What Problem Does This Paper Address?

Evaluating causal effect estimators is notoriously difficult because real-world interventions are often infeasible, and observational data requires strong, often untestable, assumptions. While simulators offer a controlled environment, existing benchmarks are frequently too simplistic to capture the nuances of real applications. This limits our ability to reliably assess how well these estimators perform in complex scenarios, particularly in healthcare where understanding treatment effects is crucial for patient care. The authors aim to address this by creating a more realistic and sophisticated simulator specifically for Alzheimer's disease.

Methods

The core of this work is the development of the Alzheimer's Disease Causal estimation Benchmark (ADCB) simulator. ADCB is built upon the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, using its longitudinal patient data to inform the simulator's structure and parameters. Key methodological aspects include:

  • Causal Graph Construction: A causal graph was designed, inspired by existing literature and verified by domain experts, to represent the relationships between various patient covariates, disease subtype, treatment, and outcomes.
  • Longitudinal Modeling: The simulator models patient trajectories over time, with variables at time $t$ depending on their parents in the causal graph at $t$ and their own previous values at $t-1$.
  • Disease Subtyping: A latent variable $Z$ is introduced to represent potential subtypes of Alzheimer's disease, inferred from the bimodal distribution of the A$\beta$ ratio in the ADNI data.
  • Treatment Simulation: Realistic treatments, inspired by common Alzheimer's drugs and their known effects from clinical trials, are simulated. Treatment assignment policies are also varied, ranging from random to covariate-based.
  • Treatment Effect Heterogeneity: The simulator models heterogeneity in treatment effects, allowing for variations based on disease subtype and controlled by a parameter $\gamma$.
  • Tunable Parameters: ADCB includes several tunable parameters (e.g., sample size $N$, overlap strength $\epsilon$, heterogeneity $\gamma$, behavior policy $\mu_B$) that allow researchers to adjust the difficulty and characteristics of the causal inference tasks.

Datasets

  • Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset: This dataset was used to fit the simulator's components, including patient covariates and outcomes.
  • ADCB Benchmark: This is a newly introduced simulator that generates longitudinal data for evaluating causal effect estimators. It aims to be more realistic than existing benchmarks like IHDP and ACIC by incorporating more complex disease dynamics and treatment effects.

Key Insights and Ideas

  • The Need for Realistic Simulators: The paper strongly emphasizes that while simulators are essential for benchmarking, their utility is directly tied to their realism. Overly simplistic simulators can lead to a false sense of confidence in causal inference methods.
  • Alzheimer's Disease Complexity: The simulator highlights key complexities in Alzheimer's disease, such as potential disease subtypes and the symptomatic, temporary nature of current treatments, which are crucial to consider for accurate causal effect estimation.
  • Tunable Difficulty: The ability to adjust parameters like sample size, overlap, and treatment effect heterogeneity allows ADCB to serve as a flexible benchmark that can systematically test the robustness of estimators under various challenging conditions.
  • Importance of History: The inclusion of longitudinal data and the comparison of single-time-point estimators with sequential learners underscore the value of temporal information in understanding causal effects in dynamic diseases.

Key Results

The paper presents experiments comparing different causal effect estimators (S-Learner, T-Learner, Sequential T-Learner) on the ADCB benchmark under various configurations.

  • PEHE Performance: The results show that the T-learner generally achieves a lower Precision in Estimation of Heterogeneous Effects (PEHE) compared to the S-learner, indicating better capture of individual-level treatment effect heterogeneity.
  • Sequential Learner Advantage: The Sequential T-Learner, which can incorporate historical data, demonstrates improved performance, especially in scenarios with more complex dynamics.
  • Parameter Impact: The figures illustrate how parameters like sample size ($N$), overlap strength ($\epsilon$), and treatment effect heterogeneity ($\gamma$) influence the performance of estimators, as expected. For instance, higher $\gamma$ (more heterogeneity) makes PEHE harder to predict.
  • Behavior Policy Influence: Different behavior policies ($\mu_B$) also impact performance, highlighting the importance of considering how treatments are assigned in observational settings.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while ADCB is more realistic than previous benchmarks, it is still a simplification of the true complexity of Alzheimer's disease.
  • The current implementation focuses on a binary disease subtype, and future work could explore more granular subtypes.
  • The treatment effects are assumed to be stationary in time, which might not hold true for all treatments or disease stages.
  • Future Work:
    • The authors plan to explore more complex confounding models in future iterations of the simulator to further increase its difficulty.
  • ADCB is intended to be used for evaluating decision-making policies and causally adaptive treatment strategies, which will be a focus of future research.
  • Expanding the range of simulated treatments and their effects is also a potential avenue.

Summary updated

Abstract

Simulators make unique benchmarks for causal effect estimation since they do not rely on unverifiable assumptions or the ability to intervene on real-world systems, but are often too simple to capture important aspects of real applications. We propose a simulator of Alzheimer's disease aimed at modeling intricacies of healthcare data while enabling benchmarking of causal effect and policy estimators. We fit the system to the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and ground hand-crafted components in results from comparative treatment trials and observational treatment patterns. The simulator includes parameters which alter the nature and difficulty of the causal inference tasks, such as latent variables, effect heterogeneity, length of observed history, behavior policy and sample size. We use the simulator to compare estimators of average and conditional treatment effects.
BibTeX
@misc{kinyanjui2021adcb,
  author = {Newton Mwai Kinyanjui and Fredrik D. Johansson},
  title = {{ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects}},
  year = {2021},
  eprint = {2111.06811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.06811},
}
Figure