Robustly Extracting Medical Knowledge from EHRs: A Case Study of Learning a Health Knowledge Graph

Irene Y. Chen, Monica Agrawal, Steven Horng, David A. Sontag

Abstract

Automating the construction of health knowledge graphs—networks that link diseases to their associated symptoms—promises to reduce the labor-intensive nature of manual curation. This paper evaluates various modeling approaches, from simple baselines to complex causal non-linear methods, using millions of Electronic Health Records (EHRs). Our findings indicate that simpler probabilistic models like "Noisy OR" often outperform more complex alternatives. Furthermore, we identify patient comorbidities and data granularity as primary factors influencing model robustness and accuracy.

Introduction

Clinicians have long relied on diagnostic tools that map symptoms to diseases. Historically, these tools, such as the Quick Medical Reference, were built through thousands of hours of expert labor. While valuable, they are expensive to maintain and often fail to generalize across different patient populations. Electronic Health Records (EHRs) provide a massive repository of real-world data that could theoretically be used to learn these relationships algorithmically. However, EHR data is notoriously messy and biased. This study investigates the robustness of medical knowledge extraction, identifying where automated methods succeed and where they falter.

Methodology

The researchers compared three distinct modeling paradigms to identify causal links between diseases and symptoms:

  1. Baseline Models: Logistic Regression (LR) and Naive Bayes (NB).
  2. Noisy OR: A probabilistic model tailored for "multiple causes, one effect" scenarios, a staple in medical AI.
  3. Causal Non-linear Methods: A novel approach utilizing the do-operator from causal inference. By using models like Random Forests (RF), the authors estimated the probability of a symptom occurring if a disease were "intervened" upon (forced to be present vs. absent).

Performance was measured against the Google Health Knowledge Graph (GHKG), an expert-verified dataset, using Precision, Recall, and the Area Under the Precision-Recall Curve (AUPRC).

Data Sources

The study utilized two large datasets from Beth Israel Deaconess Medical Center:

  • ED Dataset: 273,174 de-identified Emergency Department records (2008–2013).
  • CR (Complete Records) Dataset: A longitudinal dataset of 140,446 patients with over 7.4 million clinical notes, processed at three levels of granularity: single notes, 30-day episodes, and full patient history.

Key Insights: The Comorbidity Confounder

A significant finding was that model complexity does not necessarily translate to better performance. Instead, the "Noisy OR" model remained the most robust across various settings. The primary obstacle to accuracy was not the sample size, but the prevalence of comorbidities. When patients present with multiple diseases, models struggle to disentangle which disease is responsible for which symptom.

Performance distribution and the impact of patient comorbidities on F1 scores.

Additionally, data granularity played a crucial role. As models moved from analyzing single notes to full patient histories, Logistic Regression improved due to increased signal, while Naive Bayes suffered as its conditional independence assumptions were violated.

Results and Performance Analysis

The Noisy OR model achieved the highest overall AUPRC of 0.3400 on the ED dataset. Interestingly, adding demographic features like age and gender did not significantly boost overall accuracy, though it provided benefits for specific age-related conditions.

AUPRC performance across different models and demographic subgroups.

Error analysis revealed that 66% of the bottom-performing diseases were associated with identifiable issues like high comorbidity or low sample size, compared to 40% in the top-performing group. Notably, the Random Forest causal method outperformed Noisy OR in only one specific case: breast cancer.

Conclusion and Future Work

This research highlights that for extracting medical knowledge from EHRs, simplicity and data quality often outweigh model complexity. Future work should address the current bipartite assumption (disease-to-symptom) by allowing for more complex disease-to-disease relationships and testing these automated graphs in live clinical environments to evaluate their real-world utility.

Summary updated

BibTeX
@inproceedings{chen2020robustly,
  author = {Irene Y. Chen and Monica Agrawal and Steven Horng and David A. Sontag},
  title = {{Robustly Extracting Medical Knowledge from EHRs: A Case Study of Learning a Health Knowledge Graph}},
  booktitle = {PSB},
  year = {2020},
  url = {https://openreview.net/forum?id=yRaIgThU-L},
}
Figure