Generating synthetic electronic health record data: a methodological scoping review with benchmarking on phenotype data and open-source software

Xingran Chen, Zhenke Wu, Xu Shi, Hyunghoon Cho, Bhramar Mukherjee

Introduction

Electronic Health Records (EHRs) are a cornerstone of modern biomedical research, providing deep insights into patient outcomes and clinical pathways. However, the utility of this data is often restricted by stringent privacy regulations and the ethical necessity of patient consent. Synthetic EHR data—computationally generated records that mimic the statistical properties of real patient data without containing information from actual individuals—has emerged as a promising solution to these accessibility challenges.

Despite the proliferation of generative models, the field remains fragmented. Researchers often face a "Wild West" of methodologies with little standardization in evaluation metrics or cross-dataset validation. This paper addresses these gaps by providing a comprehensive scoping review of existing methods and introducing SynthEHRella, an open-source Python toolkit designed for the rigorous benchmarking of synthetic EHR generation.

Methodological Landscape

The authors conducted a scoping review of 48 studies, categorizing the current state of the art into five primary methodological families:

  1. Rule-based Models: These utilize predefined clinical algorithms and census data to simulate patient lifecycles (e.g., Synthea).
  2. Generative Adversarial Networks (GANs): These models employ a competitive framework where a generator attempts to create realistic data while a discriminator attempts to distinguish it from real records (e.g., MedGAN, CorGAN).
  3. Variational Autoencoders (VAEs): These models compress EHR data into a lower-dimensional latent space before reconstructing it to generate new samples.
  4. Transformer-based Models: Leveraging attention mechanisms similar to Large Language Models (LLMs), these are increasingly used to capture complex dependencies in clinical sequences (e.g., PromptEHR).
  5. Diffusion-based Models: A recent innovation in generative AI, these models create data by iteratively reversing a noise process (e.g., EHRDiff).

The SynthEHRella Benchmarking Framework

To evaluate these methods, the authors developed SynthEHRella, a unified evaluation framework. The benchmarking process utilized the gold-standard open-source EHR datasets, MIMIC-III and MIMIC-IV, which contain data from hundreds of thousands of ICU admissions. To ensure clinical relevance, raw ICD diagnostic codes were mapped to the PhecodeX system, allowing for more meaningful phenotype analysis.

A critical component of this study was the evaluation of transportability. By training models on MIMIC-III (2001–2012) and evaluating them on MIMIC-IV (2008–2019), the researchers tested how well synthetic data handles the distributional shifts that naturally occur in healthcare settings over time.

Key Insights and Performance Metrics

The benchmarking process evaluated models across three primary dimensions: Fidelity, Utility, and Privacy.

Fidelity and Association

While simple baselines like Prevalence-based Random (PBR) models can accurately match the frequency of individual diseases, they fail to capture the complex correlations between them. In contrast, GAN-based methods, particularly CorGAN, demonstrated superior fidelity by preserving the intricate associations between variables (e.g., the co-occurrence of obesity and type 2 diabetes).

Downstream Utility

In "Train on Synthetic, Test on Real" (TSTR) scenarios—where a predictive model is trained on synthetic data and validated on real patient records—MedGAN emerged as the leader. It showed the smallest performance drop (only a 0.063 decrease in AUC), suggesting that its synthetic output is highly effective for training machine learning models intended for clinical application.

Efficiency and Privacy

The study highlighted a significant trade-off between privacy and utility. Rule-based methods like Synthea offer high privacy because they do not "memorize" real patient patterns, but they often struggle to replicate the messy correlations found in real-world clinical data. Furthermore, the computational efficiency varied wildly: GANs were remarkably fast, with MedGAN generating samples in fractions of a second, while some rule-based simulators took several hours to produce comparable cohorts.

Conclusion and Future Directions

The research concludes that while GANs currently offer the most balanced performance for cross-sectional phenotype data, the field is rapidly evolving. Newer architectures like Diffusion and Transformers show significant promise for longitudinal data—tracking patient trajectories over time—which remains a complex challenge for current generators.

Future work must address the limitations of current benchmarks, including the handling of missing data, the integration of multimodal sources (such as clinical notes and medical imaging), and the development of conditional generation methods to ensure fairness and representation across diverse demographic groups. By providing SynthEHRella to the community, the authors have established a foundation for more transparent and reproducible synthetic EHR research.

Summary updated

Abstract

We conduct a scoping review of existing approaches for synthetic EHR data generation, and benchmark major methods with proposed open-source software to offer recommendations for practitioners. We search three academic databases for our scoping review. Methods are benchmarked on open-source EHR datasets, MIMIC-III/IV. Seven existing methods covering major categories and two baseline methods are implemented and compared. Evaluation metrics concern data fidelity, downstream utility, privacy protection, and computational cost. 42 studies are identified and classified into five categories. Seven open-source methods covering all categories are selected, trained on MIMIC-III, and evaluated on MIMIC-III or MIMIC-IV for transportability considerations. Among them, GAN-based methods demonstrate competitive performance in fidelity and utility on MIMIC-III; rule-based methods excel in privacy protection. Similar findings are observed on MIMIC-IV, except that GAN-based methods further outperform the baseline methods in preserving fidelity. A Python package, "SynthEHRella", is provided to integrate various choices of approaches and evaluation metrics, enabling more streamlined exploration and evaluation of multiple methods. We found that method choice is governed by the relative importance of the evaluation metrics in downstream use cases. We provide a decision tree to guide the choice among the benchmarked methods. Based on the decision tree, GAN-based methods excel when distributional shifts exist between the training and testing populations. Otherwise, CorGAN and MedGAN are most suitable for association modeling and predictive modeling, respectively. Future research should prioritize enhancing fidelity of the synthetic data while controlling privacy exposure, and comprehensive benchmarking of longitudinal or conditional generation methods.
BibTeX
@article{chen2025generating,
  author = {Xingran Chen and Zhenke Wu and Xu Shi and Hyunghoon Cho and Bhramar Mukherjee},
  title = {{Generating synthetic electronic health record data: a methodological scoping review with benchmarking on phenotype data and open-source software}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2025},
  doi = {10.1093/jamia/ocaf082},
}
Figure