The National COVID Cohort Collaborative: Analyses of Original and Computationally Derived Electronic Health Record Data (Preprint)

Randi E. Foraker, Aixia Guo, Jason Thomas, Noa Zamstein, Philip Payne, Adam Wilcox, N3c Collaborative2021 · DOI

TL;DR

This paper validates the use of synthetic electronic health record (EHR) data generated by MDClone for COVID-19 research. The study found that synthetic data closely mimics original data across various analytical tasks, including exploring cohort distributions, training predictive models, and generating epidemic curves, thereby accelerating collaborative research while preserving patient privacy.

What Problem Does This Paper Address?

Sharing sensitive patient data for large-scale research, especially during a public health crisis like COVID-19, is fraught with privacy concerns and regulatory hurdles. Traditional methods of data anonymization can sometimes strip away valuable information or introduce biases. The National COVID Cohort Collaborative (N3C) aims to overcome these challenges by leveraging synthetic data, which is computationally generated to retain the statistical properties of the original data without containing actual patient information. This paper addresses the critical question of whether this synthetic data is a reliable substitute for original EHR data, specifically within the context of COVID-19 research, and assesses its strengths and limitations for collaborative efforts.

Methods

The researchers utilized the National COVID Cohort Collaborative's (N3C) instance of MDClone, a platform designed for data synthesis. They conducted three distinct use cases to compare analyses performed on original data versus synthetic data:

  1. Exploring Cohort Distributions: They examined the distributions of key demographic and clinical features (e.g., age, gender, race, comorbidities) for COVID-19 positive patients.
  2. Predictive Modeling: They trained and tested machine learning models (logistic regression and random forest) to predict the risk of hospital admission for COVID-19 patients.
  3. Geospatial and Temporal Analysis: They constructed epidemic curves using geospatial (zip code) and temporal data to track COVID-19 trends.

Analyses involved traditional statistics, machine learning approaches, and temporal/spatial representations. Comparisons were made using metrics like means, standard deviations, odds ratios, and model performance scores (accuracy, ROC AUC, precision-recall).

Datasets

The study used data from the National COVID Cohort Collaborative (N3C), which comprises EHR data from 34 institutional partners across the United States. Both original and computationally derived (synthetic) versions of this multi-institutional dataset were used for the analyses. No new benchmarks were introduced; rather, the paper focuses on validating the synthetic data against the existing original N3C data.

Key Insights and Ideas

  • Synthetic Data as a Viable Alternative: The core finding is that synthetic data can effectively replicate the statistical properties and analytical outcomes of original EHR data for many research purposes. This is crucial for enabling data sharing and accelerating research without compromising patient privacy.
  • Privacy-Preserving Research Acceleration: The ability to use synthetic data means researchers can access and analyze large datasets more rapidly, bypassing some of the lengthy data use agreements and privacy reviews associated with original data.
  • Limitations in Low-Population Areas: A significant limitation emerged in the geospatial analysis, where synthetic data showed discrepancies in areas with low population counts. This is attributed to data suppression (censoring) in synthetic data to protect privacy, which can disproportionately affect smaller geographic regions and potentially mask data quality issues in those areas.
  • Temporal Validity: The study suggests that synthetic data can maintain temporal validity, offering an alternative to date-shifting techniques for privacy protection while preserving time-series analysis capabilities.

Key Results

  • Demographic and Clinical Distributions: For Use Case 1, the distributions of demographic and clinical variables were nearly identical between the original and synthetic datasets. For example, mean age and gender distribution were the same.
  • Predictive Model Performance: In Use Case 2, predictive models trained and tested on synthetic data showed comparable performance to those trained on original data. For instance, the Area Under the ROC Curve (AUC) for the random forest model was 0.814 for original data and 0.812 for synthetic data. Odds ratios for admission risk factors were also very similar.
  • Epidemic Curve Concordance: For Use Case 3, epidemic curves generated from synthetic data closely mirrored those from original data, with no statistically significant differences found in aggregate counts, 7-day moving averages, or 7-day slopes.
  • Notable Exceptions: A minor exception was noted in multivariable analyses where an odds ratio fell on opposite sides of the null (0.97 vs 1.01). Additionally, differences in epidemic curve magnitudes were observed for zip codes with low population counts due to data censoring.

Limitations and Future Work

The authors acknowledge several limitations:

  • Privacy Evaluation: The study focused on statistical similarity and did not include a formal privacy evaluation of the synthetic data.
  • P-value Threshold: A standard p-value threshold of 0.05 was used, which may not be suitable for all use cases, and equivalence testing could be a more appropriate statistical approach in some scenarios.
  • Geospatial Analysis Scope: The geospatial analysis was limited in scope, and future work will explore other measures and finer-grained geographic levels.

Future work suggested includes:

  • Developing and validating additional clinical risk prediction models.
  • Employing a broader range of analytic methods.
  • Conducting more detailed geospatial and temporal analyses.
  • Continuously validating synthetic data for various use cases and analytic methods.
  • Exploring the performance of synthetic data in commercial systems and for software engineering projects.

Summary updated

Abstract

<sec> <title>BACKGROUND</title> Computationally derived (“synthetic”) data can enable the creation and analysis of clinical, laboratory, and diagnostic data as if they were the original electronic health record data. Synthetic data can support data sharing to answer critical research questions to address the COVID-19 pandemic. </sec> <sec> <title>OBJECTIVE</title> We aim to compare the results from analyses of synthetic data to those from original data and assess the strengths and limitations of leveraging computationally derived data for research purposes. </sec> <sec> <title>METHODS</title> We used the National COVID Cohort Collaborative’s instance of MDClone, a big data platform with data-synthesizing capabilities (MDClone Ltd). We downloaded electronic health record data from 34 National COVID Cohort Collaborative institutional partners and tested three use cases, including (1) exploring the distributions of key features of the COVID-19–positive cohort; (2) training and testing predictive models for assessing the risk of admission among these patients; and (3) determining geospatial and temporal COVID-19–related measures and outcomes, and constructing their epidemic curves. We compared the results from synthetic data to those from original data using traditional statistics, machine learning approaches, and temporal and spatial representations of the data. </sec> <sec> <title>RESULTS</title> For each use case, the results of the synthetic data analyses successfully mimicked those of the original data such that the distributions of the data were similar and the predictive models demonstrated comparable performance. Although the synthetic and original data yielded overall nearly the same results, there were exceptions that included an odds ratio on either side of the null in multivariable analyses (0.97 vs 1.01) and differences in the magnitude of epidemic curves constructed for zip codes with low population counts. </sec> <sec> <title>CONCLUSIONS</title> This paper presents the results of each use case and outlines key considerations for the use of synthetic data, examining their role in collaborative research for faster insights. </sec>
BibTeX
@misc{foraker2021national,
  author = {Randi E. Foraker and Aixia Guo and Jason Thomas and Noa Zamstein and Philip Payne and Adam Wilcox and N3C Collaborative},
  title = {{The National COVID Cohort Collaborative: Analyses of Original and Computationally Derived Electronic Health Record Data (Preprint)}},
  year = {2021},
  doi = {10.2196/preprints.30697},
}
Figure