Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: Results from analyzing >1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)

Jason Thomas, Randi E. Foraker, Noa Zamstein, Philip Payne, Adam WilcoxmedRxiv 2021 · DOI

Abstract

In general, synthetic data were successfully used to analyze geospatial and temporal trends. Analyses using small sample sizes or populations were limited, in part due to purposeful data label suppression—an attribute disclosure countermeasure. Users should consider data fitness for use in these cases.

Introduction

During a pandemic, rapid access to granular, individual-level Electronic Health Record (EHR) data is critical for understanding viral spread. However, sharing such data presents significant privacy risks. While the National COVID Cohort Collaborative (N3C) centralizes this information, access remains restricted even within secure enclaves. Synthetic data—computationally generated to mimic the statistical properties of original records without containing actual patient data—offers a potential solution. This study explores the privacy-utility trade-off: determining if synthetic data can maintain enough utility for local epidemiological tracking while protecting individual privacy, particularly in sparsely populated areas.

Methods

Researchers conducted a side-by-side "twinning" analysis comparing an original Limited Data Set (LDS) of 1,854,968 SARS-CoV-2 tests against a synthetic version generated via the MDClone platform. Utility was assessed through:

  1. Aggregate Curves: Constructing national epidemic curves for tests, cases, hospitalizations, and deaths.
  2. Geospatial Analysis: Evaluating data at the 5-digit zip code level.
  3. Statistical Validation: Employing paired t-tests and Wilcoxon signed-rank tests to compare synthetic and real-world curves.
  4. Error Stratification: Calculating "synthetic error" across various timeframes and zip code densities.

Data Sources

The study utilized the N3C Data Enclave, incorporating EHR data from 34 source partners across the U.S. The dataset included approximately 1.85 million tests with features such as test results, age, hospital admission dates, length of stay (LOS), and mortality. This was further enriched with Social Determinants of Health (SDOH) data at the zip code level, including poverty metrics and insurance status.

Results

The analysis revealed high fidelity at scale. National epidemic curves for tests, cases, and deaths showed no statistically significant differences between the original and synthetic datasets (p-values > 0.05). Demographic alignment was also strong; for instance, the mean age in both datasets was 44. In high-density urban areas (the top 1% of most-tested zip codes), the synthetic data was nearly indistinguishable from the original.

Table 1: Statistical comparison of original and synthetic datasets across test outcomes and demographics.

However, a significant "censorship" of rural data occurred. To protect privacy, the generator suppressed zip codes with very few tests (mean of 2.9), leading to a 66% reduction in unique zip codes (from 17,041 to 5,819). Additionally, a "non-negativity bias" was observed, where the synthetic model slightly overestimated counts in low-density areas and underestimated them in high-density areas.

Discussion and Limitations

One unexpected benefit of synthetic generation was "accidental data cleaning." The process attenuated extreme outliers, such as an erroneous 120-year hospital stay in the original data, resulting in more realistic mean values.

Despite its utility for broad public health analysis, synthetic data has limitations. This study focused on utility rather than privacy, and did not perform adversarial re-identification attacks. Furthermore, the synthetic data was produced in a model different from the standard OMOP Common Data Model used by N3C, requiring code modifications for researchers moving from synthetic to restricted data. Future research should focus on equivalence testing and improving the representation of rural populations without compromising privacy.

Summary updated

Abstract

In general, synthetic data were successfully used to analyze geospatial and temporal trends. Analyses using small sample sizes or populations were limited, in part due to purposeful data label suppression -an attribute disclosure countermeasure. Users should consider data fitness for use in these cases.
BibTeX
@misc{thomas2021demonstrating,
  author = {Jason Thomas and Randi E. Foraker and Noa Zamstein and Philip Payne and Adam Wilcox},
  title = {{Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: Results from analyzing \>1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)}},
  year = {2021},
  doi = {10.1101/2021.07.06.21259051},
  howpublished = {medRxiv},
}
Figure