TL;DR
This paper introduces KEEPER, a novel system that leverages standardized structured electronic health record (EHR) data to perform phenotype evaluation, offering a scalable and interpretable alternative to traditional manual chart review. KEEPER significantly reduces review time while achieving comparable accuracy, paving the way for more efficient and widespread observational research.
What Problem Does This Paper Address?
Phenotype evaluation, the process of identifying specific patient groups for research based on their characteristics, is crucial for the validity of observational studies. The current gold standard is manual chart review, which involves clinicians meticulously examining patient records. However, this method is extremely time-consuming, labor-intensive, prone to variability, and often requires access to sensitive patient data that may not be available across all data sources (e.g., administrative claims data). Consequently, researchers often review only a small subset of patients, risking selection bias, or forgo evaluation altogether. This limitation hinders the scalability of observational research, especially for complex or rare conditions.
Methods
The core innovation of this paper is the development of the Knowledge-Enhanced Electronic Profile Review (KEEPER) system. KEEPER is built upon three key principles:
- Adherence to Clinical Reasoning: KEEPER structures patient data to mimic the logical steps a clinician would take during a diagnostic process. This includes organizing information into categories like clinical presentation, history, preliminary diagnosis, diagnostic procedures, treatment, and follow-up care.
- Standardization: Data extraction and representation are standardized using the Observational Medical Outcomes Partnership (OMOP) Common Data Model and standardized vocabularies. This ensures that KEEPER can be applied across different data sources and conditions without modification, making it disease-agnostic.
- Dimensionality Reduction: Instead of presenting the entire patient record, KEEPER extracts and presents only the structured data elements clinically relevant to the specific phenotype being evaluated. This significantly reduces the volume of information, making the review process more efficient.
KEEPER takes a cohort of patients identified by a phenotype definition and extracts relevant structured data elements, presenting them in a standardized profile.
Datasets
The study utilized electronic health records (EHRs) from Columbia University Irving Medical Center (CUIMC), translated into the OMOP Common Data Model. The data included demographics, visits, conditions, drugs, devices, measurements, and observations. For evaluation, the researchers used a random sample of 20 patients per condition from existing eMERGE algorithms for four conditions: acute appendicitis, diabetes mellitus type I (DM type I), chronic obstructive pulmonary disorder (COPD), and end-stage renal disease (ESRD). The "gold standard" for comparison was established through independent chart reviews by two authors, with disagreements resolved through discussion.
Key Insights and Ideas
- Mimicking Clinical Logic is Key: Organizing structured data according to the natural flow of clinical reasoning significantly enhances interpretability and efficiency, making it a powerful alternative to unstructured chart review.
- Structured Data Suffices (When Organized): The study demonstrates that structured EHR data, when curated and presented thoughtfully, can provide sufficient information for accurate phenotype evaluation, even without access to free-text notes.
- Scalability Through Standardization: The disease-agnostic nature of KEEPER, achieved through standardized data models and reasoning principles, is crucial for scaling phenotype evaluation across diverse conditions and data sources.
- Efficiency Gains are Substantial: The reduction in review time is not marginal; KEEPER offers a more than 50% speedup, enabling researchers to analyze larger cohorts or more conditions.
Key Results
- Efficiency: KEEPER reduced review time by over half. Chart review for 20 patients took an average of 67 minutes (SD=43), while KEEPER profile review took 30 minutes (SD=14) (P value .04).
- Interrater Agreement: KEEPER significantly improved consistency between reviewers. Reviewers agreed on patient status 91.2% of the time with KEEPER, compared to 76.3% with manual chart review.
- Intermethod Agreement: Overall, 88.1% of patients were classified concordantly using KEEPER and manual chart review, with a Cohen's kappa of 0.71 (substantial agreement). Agreement varied by condition, with acute appendicitis showing almost perfect agreement and DM type I showing moderate agreement.
- Accuracy: KEEPER's accuracy was noninferior to manual chart review. Overall, 88.1% of KEEPER classifications aligned with the gold standard, compared to 86.9% for manual chart review. Accuracy for KEEPER was at least 80% for all tested conditions.
Limitations and Future Work
- Data Source Specificity: The study was conducted on rich data from a tertiary healthcare center. Findings may not generalize to institutions with higher data loss from charts to structured records or to data sources with different data elements (e.g., lacking pharmacy claims).
- Limited Conditions: The experiment was performed on four conditions. Generalizability to other conditions, especially those with low healthcare utilization or asymptomatic presentations, needs further investigation.
- Unexamined Claims Data: KEEPER's performance on data sources lacking charts (like administrative claims) was not explicitly examined, though it is a primary target for this approach.
- Chart Review Protocol: The study did not use fully specified chart review protocols, which could influence comparative performance.
- Gold Standard Construction: While rigorous, the gold standard was established by iterative chart review, and accuracy estimates should be interpreted with this in mind.
- Development Cost: There is an initial cost associated with developing the input (code selection) for KEEPER.
Future work should focus on developing automated, disease-agnostic methods for extracting relevant information for KEEPER to further reduce development costs and enhance scalability. Integrating KEEPER into broader OHDSI (Observational Health Data Sciences and Informatics) tools is also envisioned.
Summary updated