Real world performance of the 21st Century Cures Act population-level application programming interface

James R. Jones, Daniel Gottlieb, Andrew J. McMurry, Ashish Atreja, Pankaja M. Desai, Brian E. Dixon, Philip R. O. Payne, Anil J. Saldanha, Prabhu R. V. Shankar, Yauheni Solad, Adam B. Wilcox, Momeena S. Ali, Eugene Kang, Andrew M. Martin, Elizabeth Sprouse, David E. Taylor, Michael Terry, Vladimir Ignatov, Kenneth D. MandlJ. Am. Medical Informatics Assoc 2024 · DOI

Introduction

The 21st Century Cures Act was designed to eliminate "information blocking" in healthcare by mandating standardized APIs for population-level data exchange. Specifically, Electronic Health Record (EHR) vendors must provide a SMART/HL7 Bulk FHIR API to allow researchers and public health agencies to export data for large cohorts with "no special effort." However, while certification ensures these APIs exist, it does not measure their real-world performance. This study benchmarks early implementations of the Bulk FHIR API across five academic medical centers to evaluate if the vision of "push button population health" is being realized.

Methodology

Researchers conducted a multi-site study from April to September 2023 across five institutions:

  • Site 1: Remote-hosted Oracle Cerner.
  • Sites 2-4: A mix of on-premises and remote-hosted Epic systems.
  • Site 5: An on-premises Health Information Exchange (HIE) using a custom-built API "facade" over a relational data warehouse.

The team used an open-source Bulk FHIR Testing Suite to request data for cohorts ranging from 1,000 to over 20,000 patients, focusing on six core FHIR resource types (Patient, Encounter, Condition, DocumentReference, Observation, and MedicationRequest). To establish a baseline, they compared the Bulk FHIR API's speed against a "Crawler" that fetched records one-by-one using the standard single-patient SMART on FHIR API.

Key Results

The study revealed significant performance disparities between custom implementations and standard EHR vendor offerings.

Performance comparison of Bulk FHIR vs. SoF Crawler across different server types and sites.
  • Top Performer: The HIE Custom API (Site 5) significantly outperformed all others, processing over 141 million resources at a rate of approximately 12,000 resources/minute.
  • Vendor Performance: Oracle Cerner (Site 1) averaged ~8,000 resources/minute. In contrast, Epic sites (Sites 2-4) were the slowest and most variable, with rates ranging from 1,555 to 2,500 resources/minute.
  • Reliability and Scale: While the HIE handled massive datasets, Epic implementations often struggled with larger cohorts. Requests for 10,000+ patients frequently timed out or encountered errors. One such export took nearly two weeks, only for the security token to expire before the data could be retrieved.

Discussion and Insights

The performance gap highlights a fundamental architectural difference: the HIE’s custom API sits on a data warehouse optimized for read/export operations, whereas standard EHR APIs often run against transactional databases designed for live clinical care. This creates a performance ceiling for standard vendor implementations.

Furthermore, the "Crawler" baseline revealed a surprising paradox: in some Epic environments, fetching records one-by-one was actually faster or more reliable than using the dedicated Bulk API. This suggests that current bulk export processes may lack sufficient parallelization. Additionally, the "no special effort" mandate remains unfulfilled; sites spent an average of 65 days (and up to 119 days) just to configure and activate the APIs.

Conclusion and Future Directions

While the 21st Century Cures Act has successfully mandated the existence of population-level APIs, their real-world utility is currently hampered by performance bottlenecks and configuration hurdles. The authors recommend that the Office of the National Coordinator for Health IT (ONC) incorporate specific performance metrics into the certification process—such as the ability to export all US Core data for a population within five days. Future research should expand the vendor pool and examine how hospital operational cycles (peak vs. off-peak hours) impact data throughput.

Summary updated

BibTeX
@article{jones2024real,
  author = {James R. Jones and Daniel Gottlieb and Andrew J. McMurry and Ashish Atreja and Pankaja M. Desai and Brian E. Dixon and Philip R. O. Payne and Anil J. Saldanha and Prabhu R. V. Shankar and Yauheni Solad and Adam B. Wilcox and Momeena S. Ali and Eugene Kang and Andrew M. Martin and Elizabeth Sprouse and David E. Taylor and Michael Terry and Vladimir Ignatov and Kenneth D. Mandl},
  title = {{Real world performance of the 21st Century Cures Act population-level application programming interface}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2024},
  doi = {10.1093/jamia/ocae040},
}
Figure