Cohort design and natural language processing to reduce bias in electronic health records research

Shaan Khurshid, Christopher Reeder, Lia X. Harrington, Pulkit Singh, Gopal Sarma, Sam Freesun Friedman, Paolo Di Achille, Nathaniel Diamant, Jonathan W. Cunningham, Ashby C. Turner, Emily S. Lau, Julian S. Haimovich, Mostafa A. Al-Alusi, Xin Wang, Marcus D. R. Klarqvist, Jeffrey M. Ashburner, Christian Diedrich, Mercedeh Ghadessi, Johanna Mielke, Hanna M. Eilken, Alice McElhinney, Andrea Derix, Steven J. Atlas, Patrick T. Ellinor, Anthony A. Philippakis, Christopher D. Anderson, Jennifer E. Ho, Puneet Batra, Steven A. Lubitz

TL;DR

This paper introduces the Community Care Cohort Project (C3PO), a novel EHR-based cohort designed to mitigate bias by focusing on patients with longitudinal primary care. By combining this cohort design with deep learning-based Natural Language Processing (NLP) to recover missing vital signs, they demonstrate improved model calibration and reduced bias compared to traditional EHR sampling methods.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of patient data, offering immense potential for medical research. However, EHR data is notoriously plagued by ascertainment bias (data collected based on clinical need, not research design) and missingness (incomplete records). These biases can lead to spurious associations and poor generalizability of findings, making it difficult to trust research derived from EHRs. Traditional approaches often sample all available data, which can amplify these biases. This paper aims to address these limitations by proposing a more robust cohort design and leveraging advanced NLP techniques to improve data completeness and reduce bias.

Methods

The core of this work lies in two key innovations:

  • Cohort Design (C3PO): Instead of sampling all individuals with any data, the researchers created the Community Care Cohort Project (C3PO) by selecting individuals who received regular, longitudinal primary care. This approach aims to capture data acquired outside of acute illness, thus reducing ascertainment bias and providing a more representative sample of the general population.
  • Natural Language Processing (NLP) for Data Recovery: A significant portion of vital sign data (height, weight, blood pressure) is often buried in unstructured clinical notes. The authors developed and employed a deep learning model (Bio + Discharge Summary BERT), a contextual word embedding model, to extract these vital signs from free-text notes. This NLP approach was compared to a rule-based method, showing superior yield.

Datasets

The primary dataset used is the Mass General Brigham (MGB) multi-institutional EHR, comprising over 3.6 million individuals.

  • C3PO Cohort: A subset of 520,868 individuals aged 18-90, selected based on receiving at least two primary care visits within a 1-3 year window.
  • Convenience Samples: Two comparison cohorts were constructed from the same MGB EHR data:
  • MI/stroke Convenience Sample: Individuals with complete data for the Pooled Cohort Equations (PCE) risk score.
  • AF Convenience Sample: Individuals with complete data for the CHARGE-AF risk score.
  • These convenience samples did not require longitudinal primary care.

Key Insights and Ideas

  • Longitudinal Primary Care as a Bias Mitigation Strategy: The study strongly suggests that focusing on patients with regular primary care encounters, rather than just any encounter, can significantly reduce ascertainment bias. This is because these visits are more likely to capture routine health data, not just data related to acute illness.
  • NLP is Crucial for Unlocking EHR Potential: The paper highlights that a substantial amount of valuable information (like vital signs) is locked away in unstructured text. Advanced NLP models can effectively unlock this data, dramatically reducing missingness and improving the completeness of EHR datasets.
  • Improved Model Calibration: By using the C3PO cohort and recovering missing data, the researchers observed that established risk prediction models (PCE and CHARGE-AF) showed better calibration. This means the models' predictions more closely aligned with actual observed outcomes, suggesting more reliable and generalizable predictions.
  • The "Abrupt Rise" Phenomenon: A striking observation was the rapid increase in disease incidence shortly after the start of follow-up in the convenience samples, which was absent in C3PO. This suggests that convenience sampling might be capturing individuals with higher, pre-existing disease risk, leading to an artificial acceleration of observed outcomes.

Key Results

  • NLP Reduced Vital Sign Missingness by 31%: The deep learning NLP model successfully recovered vital sign data, significantly improving data completeness.
  • High Agreement Between NLP and Tabular Data: NLP-extracted vital signs were highly correlated with tabular data (Pearson r range 0.95-0.99) and showed good agreement, validating the NLP approach.
  • Reduced Bias in C3PO:
    • Lower Incidence Rates: C3PO showed lower rates of myocardial infarction/stroke and atrial fibrillation compared to convenience samples, suggesting reduced ascertainment bias.
  • Better Model Calibration: Risk models (PCE and CHARGE-AF) were better calibrated in C3PO, with lower calibration errors (e.g., 0.012-0.030 in C3PO vs. 0.028-0.046 in convenience samples for MI/stroke).
  • Improved Precision: While not always improving accuracy, NLP recovery led to less uncertainty in metric estimates, indicating increased statistical power.

Limitations and Future Work

  • Residual Indication Bias: Despite efforts, some degree of indication bias may still exist in EHR data.
  • Missingness in Other Features: While vital signs were addressed, missingness in other features (e.g., cholesterol) remains a challenge.
  • Generalizability of NLP: The performance of the NLP model on datasets outside of MGB needs further validation.
  • Demographic Limitations: The C3PO cohort is predominantly White, limiting generalizability to other racial compositions.
  • Observational Nature: The study's findings are observational and cannot infer causality.

Future work suggested includes:

  • Developing NLP models to extract other clinical parameters like laboratory values.
  • Investigating how NLP-based recovery compares to other missing data imputation techniques.
  • Further research into understanding and mitigating residual biases.
  • Exploring more advanced recalibration or reweighting techniques for EHR data.

Summary updated

Abstract

Electronic health record (EHR) datasets are statistically powerful but are subject to ascertainment bias and missingness. Using the Mass General Brigham multi-institutional EHR, we approximated a community-based cohort by sampling patients receiving longitudinal primary care between 2001-2018 (Community Care Cohort Project [C3PO], n = 520,868). We utilized natural language processing (NLP) to recover vital signs from unstructured notes. We assessed the validity of C3PO by deploying established risk models for myocardial infarction/stroke and atrial fibrillation. We then compared C3PO to Convenience Samples including all individuals from the same EHR with complete data, but without a longitudinal primary care requirement. NLP reduced the missingness of vital signs by 31%. NLP-recovered vital signs were highly correlated with values derived from structured fields (Pearson r range 0.95–0.99). Atrial fibrillation and myocardial infarction/stroke incidence were lower and risk models were better calibrated in C3PO as opposed to the Convenience Samples (calibration error range for myocardial infarction/stroke: 0.012–0.030 in C3PO vs. 0.028–0.046 in Convenience Samples; calibration error for atrial fibrillation 0.028 in C3PO vs. 0.036 in Convenience Samples). Sampling patients receiving regular primary care and using NLP to recover missing data may reduce bias and maximize generalizability of EHR research.
BibTeX
@article{khurshid2022cohort,
  author = {Shaan Khurshid and Christopher Reeder and Lia X. Harrington and Pulkit Singh and Gopal Sarma and Sam Freesun Friedman and Paolo Di Achille and Nathaniel Diamant and Jonathan W. Cunningham and Ashby C. Turner and Emily S. Lau and Julian S. Haimovich and Mostafa A. Al-Alusi and Xin Wang and Marcus D. R. Klarqvist and Jeffrey M. Ashburner and Christian Diedrich and Mercedeh Ghadessi and Johanna Mielke and Hanna M. Eilken and Alice McElhinney and Andrea Derix and Steven J. Atlas and Patrick T. Ellinor and Anthony A. Philippakis and Christopher D. Anderson and Jennifer E. Ho and Puneet Batra and Steven A. Lubitz},
  title = {{Cohort design and natural language processing to reduce bias in electronic health records research}},
  journal = {Npj Digit. Medicine},
  year = {2022},
  doi = {10.1038/s41746-022-00590-0},
}
Figure