TL;DR
This paper introduces the Community Care Cohort Project (C3PO), a novel EHR-based cohort designed to mitigate bias by focusing on patients with longitudinal primary care. By combining this cohort design with deep learning-based Natural Language Processing (NLP) to recover missing vital signs, they demonstrate improved model calibration and reduced bias compared to traditional EHR sampling methods.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) are a treasure trove of patient data, offering immense potential for medical research. However, EHR data is notoriously plagued by ascertainment bias (data collected based on clinical need, not research design) and missingness (incomplete records). These biases can lead to spurious associations and poor generalizability of findings, making it difficult to trust research derived from EHRs. Traditional approaches often sample all available data, which can amplify these biases. This paper aims to address these limitations by proposing a more robust cohort design and leveraging advanced NLP techniques to improve data completeness and reduce bias.
Methods
The core of this work lies in two key innovations:
- Cohort Design (C3PO): Instead of sampling all individuals with any data, the researchers created the Community Care Cohort Project (C3PO) by selecting individuals who received regular, longitudinal primary care. This approach aims to capture data acquired outside of acute illness, thus reducing ascertainment bias and providing a more representative sample of the general population.
- Natural Language Processing (NLP) for Data Recovery: A significant portion of vital sign data (height, weight, blood pressure) is often buried in unstructured clinical notes. The authors developed and employed a deep learning model (Bio + Discharge Summary BERT), a contextual word embedding model, to extract these vital signs from free-text notes. This NLP approach was compared to a rule-based method, showing superior yield.
Datasets
The primary dataset used is the Mass General Brigham (MGB) multi-institutional EHR, comprising over 3.6 million individuals.
- C3PO Cohort: A subset of 520,868 individuals aged 18-90, selected based on receiving at least two primary care visits within a 1-3 year window.
- Convenience Samples: Two comparison cohorts were constructed from the same MGB EHR data:
- MI/stroke Convenience Sample: Individuals with complete data for the Pooled Cohort Equations (PCE) risk score.
- AF Convenience Sample: Individuals with complete data for the CHARGE-AF risk score.
- These convenience samples did not require longitudinal primary care.
Key Insights and Ideas
- Longitudinal Primary Care as a Bias Mitigation Strategy: The study strongly suggests that focusing on patients with regular primary care encounters, rather than just any encounter, can significantly reduce ascertainment bias. This is because these visits are more likely to capture routine health data, not just data related to acute illness.
- NLP is Crucial for Unlocking EHR Potential: The paper highlights that a substantial amount of valuable information (like vital signs) is locked away in unstructured text. Advanced NLP models can effectively unlock this data, dramatically reducing missingness and improving the completeness of EHR datasets.
- Improved Model Calibration: By using the C3PO cohort and recovering missing data, the researchers observed that established risk prediction models (PCE and CHARGE-AF) showed better calibration. This means the models' predictions more closely aligned with actual observed outcomes, suggesting more reliable and generalizable predictions.
- The "Abrupt Rise" Phenomenon: A striking observation was the rapid increase in disease incidence shortly after the start of follow-up in the convenience samples, which was absent in C3PO. This suggests that convenience sampling might be capturing individuals with higher, pre-existing disease risk, leading to an artificial acceleration of observed outcomes.
Key Results
- NLP Reduced Vital Sign Missingness by 31%: The deep learning NLP model successfully recovered vital sign data, significantly improving data completeness.
- High Agreement Between NLP and Tabular Data: NLP-extracted vital signs were highly correlated with tabular data (Pearson r range 0.95-0.99) and showed good agreement, validating the NLP approach.
- Reduced Bias in C3PO:
- Lower Incidence Rates: C3PO showed lower rates of myocardial infarction/stroke and atrial fibrillation compared to convenience samples, suggesting reduced ascertainment bias.
- Better Model Calibration: Risk models (PCE and CHARGE-AF) were better calibrated in C3PO, with lower calibration errors (e.g., 0.012-0.030 in C3PO vs. 0.028-0.046 in convenience samples for MI/stroke).
- Improved Precision: While not always improving accuracy, NLP recovery led to less uncertainty in metric estimates, indicating increased statistical power.
Limitations and Future Work
- Residual Indication Bias: Despite efforts, some degree of indication bias may still exist in EHR data.
- Missingness in Other Features: While vital signs were addressed, missingness in other features (e.g., cholesterol) remains a challenge.
- Generalizability of NLP: The performance of the NLP model on datasets outside of MGB needs further validation.
- Demographic Limitations: The C3PO cohort is predominantly White, limiting generalizability to other racial compositions.
- Observational Nature: The study's findings are observational and cannot infer causality.
Future work suggested includes:
- Developing NLP models to extract other clinical parameters like laboratory values.
- Investigating how NLP-based recovery compares to other missing data imputation techniques.
- Further research into understanding and mitigating residual biases.
- Exploring more advanced recalibration or reweighting techniques for EHR data.
Summary updated