TL;DR
This study investigates how different ways of defining medical conditions (phenotypes) in electronic health records affect estimated disease incidence rates across a global network of data sources. The findings highlight that even seemingly minor changes in phenotype definitions, particularly regarding the setting of care (inpatient vs. outpatient), can significantly alter incidence rate estimates, underscoring the critical need for careful definition evaluation in observational research.
What Problem Does This Paper Address?
In medical research, especially for vaccine safety surveillance, accurately estimating the background incidence rate of adverse events is crucial. This involves defining specific medical conditions (phenotypes) within large datasets like electronic health records (EHRs) or claims data. However, there's often variability in how these phenotypes are defined, leading to inconsistent and potentially misleading incidence rates across different studies or data sources. This paper tackles the challenge of understanding how these definitional choices impact observed incidence rates, particularly in the context of COVID-19 vaccine safety monitoring, where rapid and reliable data analysis is paramount. The core problem is the lack of standardized methods for evaluating phenotype definitions and their downstream effects on incidence rate estimation in a global, multi-database setting.
Methods
The researchers employed a systematic approach to evaluate the impact of three common phenotype modification strategies on incidence rates. They used a network of 16 data sources from 10 countries, harmonized to the Observational Medical Outcomes Partnership (OMOP) Common Data Model.
The three modification types explored were:
- Inpatient Restriction: Comparing definitions that only consider events occurring in an inpatient setting versus those that include events from any place of service (inpatient, emergency room, outpatient).
- Standardization: Evaluating the effect of using standardized vocabularies (like SNOMED-CT) versus source-specific codes (like ICD-10-CM).
- Code Set Change: Examining how adding or modifying codes within a definition (e.g., including sequela codes or cause-specific codes) affects the incidence rate.
They calculated incidence rates per 100,000 person-years for 13 adverse events of special interest (AESIs) and then computed rate ratios (RRs) to compare the modified definitions against a "base" definition. These analyses were stratified by age and sex. A sensitivity analysis was also performed on the Guillain-Barré Syndrome (GBS) phenotype to explore baseline characteristics across different definitions.
Datasets
The study utilized 16 real-world data sources from 10 countries, comprising both claims data and electronic health records (EHRs). These data sources were harmonized to the OMOP Common Data Model.
The data sources included:
- Claims Data: JMDC (Japan), IBM MarketScan Commercial Claims and Encounters Database (CCAE_US), IBM MarketScan Medicare Supplemental Database (MDCR_US), IBM MarketScan Multi-State Medicaid Database (MDCD_US), and Optum De-Identified Clinformatics Extended Data Mart Database (OPTUM_DOD_US).
- EHR Data: IQVIA Australia Longitudinal Patient Data (IQVIA_AUSTRALIA), Integrated Primary Care Information (IPCI_NETHERLANDS), IQVIA Disease Analyzer (DA) Germany (IQVIA_GERMANY), Clinical Practice Research Datalink (CPRD_UK), Columbia University Irving Medical Center (CUMC_US), Optum de-identified Electronic Health Record Dataset (OPTUM_EHR_US), Health Data Warehouse of Assistance Publique-Hôpitaux de Marseille (APHM_FRANCE), Information System of Parc Salut Mar Barcelona (PSMAR_SPAIN), University Clinical Center of Serbia (CC_SERBIA), Health Informatics Centre from University of Dundee (HIC_SCOTLAND), and UK Biobank (BIOBANK_UK).
Key Insights and Ideas
- Inpatient Restriction is a Major Driver of Variation: The most significant impact on incidence rates came from restricting definitions to inpatient settings. This suggests that the choice of where an event is recorded (hospital vs. clinic) can dramatically inflate or deflate observed rates, likely due to differences in data capture and the nature of the events themselves.
- Standardization Offers Limited Impact on Incidence Rates: Interestingly, standardizing code sets (using common vocabularies like SNOMED-CT) showed minimal impact on incidence rates compared to using source-specific codes. This implies that while standardization is crucial for data harmonization, it might not be the primary driver of incidence rate discrepancies when comparing across diverse data sources.
- Code Set Granularity Matters: Modifying code sets, particularly by expanding them to include more specific or related conditions, can lead to substantial changes in incidence rates, often with considerable heterogeneity across age and sex. This highlights the trade-off between capturing a broader range of events and maintaining a consistent definition.
- Network Analysis Uncovers Hidden Heterogeneity: Evaluating phenotype definitions across a global network of diverse data sources is essential. What might appear consistent within a single database can reveal significant variability when aggregated across multiple sources with different data structures and coding practices.
- Phenotype Definition is Not Trivial: The study strongly emphasizes that phenotype definition is not a simple, one-size-fits-all task. The choices made can profoundly influence research findings, making thorough evaluation and justification of definitions critical for reliable observational studies.
Key Results
- Inpatient Restriction: Rate ratios (RRs) for inpatient restriction varied widely, ranging from 1 to 11.93. Appendicitis showed the highest variation (1-11.93), while GBS had the lowest (1.05-3.05). Claims data generally showed less variance than EHRs for this modification.
- Standardization: Standardization of code sets resulted in RRs ranging from 1 to 1.64. Pulmonary embolism had the highest RR (1.64), while anaphylaxis showed the smallest change (highest RR of 1.12).
- Code Set Change: Changes to code sets yielded RRs ranging from 1 to 2.52. Hemorrhagic stroke had the highest RR (2.73), followed by myocarditis (2.24). Most phenotypes showed significant heterogeneity by age and gender with this modification.
- GBS Sensitivity Analysis: For GBS, restricting to inpatient settings increased incidence rates by at least a 2-fold factor compared to the base definition. Further restricting to "inpatient primary position" led to even higher increases.
Limitations and Future Work
- Lack of Performance Metrics: The study focused on incidence rate changes but did not directly evaluate the sensitivity and specificity of the different phenotype definitions. The authors suggest that incorporating these performance characteristics would be beneficial for selecting optimal phenotypes.
- Data Availability: Not all data sources had complete information for all types of analyses (e.g., inpatient data was unavailable in some EHR sources), leading to missing data for certain comparisons.
- Evolving Vocabularies: Standard vocabularies can change over time, necessitating periodic re-evaluation of phenotype definitions to ensure they accurately reflect the intended population and concepts.
- Future Work: The authors advocate for continued systematic evaluation of phenotype definitions across data networks. They suggest dedicating more resources to exploring the impact of definitions before conducting large-scale studies. Future work could also involve developing methods to assess phenotype performance characteristics (sensitivity, specificity) in a federated manner across networks. The study also highlights the need to consider how data is recorded and potential coding errors as sources of variation.
Summary updated