Establishing the Automatic Identification of Clinical Trial Cohorts from Electronic Health Records by Matching Normalized Eligibility Criteria and Patient Clinical Characteristics

K. Lee, Yun Mai, Z. Liu, Kalpana Raja, Tomi Jun, Meng Ma, Tao Wang, Lei Ai, Ediz S. Calay, William Oh, Eric E. Schadt, Xingzhu WangmedRxiv 2024 · DOI

This paper describes a new approach to a long-standing bottleneck in medical research: finding the right patients for clinical trials. By creating a pipeline that "translates" messy hospital records and complex trial requirements into a shared, computable language, researchers achieved a high-accuracy matching system that could significantly accelerate the recruitment process.

TL;DR

Researchers developed an automated pipeline to match patients from Electronic Health Records (EHRs) to clinical trial eligibility criteria with a high 0.94 F1 score. The system bridges the gap between unstructured trial requirements and diverse clinical data by using a combination of standard medical terminologies and custom, in-house knowledge bases.

What Problem Does This Paper Address?

Clinical trials are the backbone of medical progress, yet many fail simply because they cannot recruit enough participants in time. The "eligibility criteria" (EC) for these trials are often written in complex, unstructured natural language (e.g., "must have failed two prior lines of therapy including an EGFR inhibitor").

On the other side, Electronic Health Records (EHRs) contain the necessary patient data, but this data is often fragmented, inconsistently coded, or buried in text. Manually reviewing thousands of patient charts to see if they match a trial's specific requirements is slow, expensive, and prone to error. The authors sought to automate this "phenotyping" process to make cohort identification faster and more precise.

Methods

The authors built a three-stage clinical phenotyping pipeline:

  1. Rule-based Knowledge Engineering: They extracted eligibility attributes from trials and categorized them into ten domains (such as condition, procedure, lab test, and therapy). For therapy-related data, they further classified items into categories like "mechanism of action" (MOA) or "regimen."
  2. Normalization: This is the core of the work. They mapped natural language terms to international standards like ICD-9/10 (for diagnoses), CPT-4 (for procedures), and LOINC (for lab tests). Crucially, they developed four in-house knowledge bases to handle concepts that standard codes miss, such as mapping a general class of drugs (e.g., "anti-androgens") to a specific list of medications.
  3. Knowledge Base Construction: All normalized attributes and patient characteristics were stored in a Redshift database, allowing for efficient SQL-based querying to match patients to trials.

Datasets

The study utilized two primary data sources:

  • Clinical Trial Criteria: Eligibility data from 3,475 clinical trials sourced from ClinicalTrials.gov, covering conditions ranging from various cancers (NSCLC, breast, prostate) to Crohn’s disease and sickle cell anemia.
  • Patient Records: EHR data from the Sema4 data warehouse, which includes the Mount Sinai Data Warehouse (MSDW) and VieCure. This dataset included demographics, vital signs, medications, and lab results.

Key Insights and Ideas

  • The "Standardization Gap": One of the most important findings was that standard medical terminologies (like ICD or LOINC) aren't enough. Only about 47% of the attributes could be normalized using standard codes alone. The remaining 53% required custom reference tables to bridge the gap between how a trial is described and how a doctor records data.
  • Domain Concentration: The researchers found that the vast majority of trial requirements (87.74%) focus on just three domains: the patient's condition, their lab tests, and their prior therapies. Prioritizing these three areas yields the highest return on investment for automation.
  • Hierarchical Mapping: The system doesn't just look for exact matches. It uses rules to map criteria at different levels of specificity. For example, a trial looking for "interstitial lung disease" can be automatically mapped to more specific ICD-10 codes like "pulmonary edema" or "acute respiratory distress syndrome."

Key Results

  • High Accuracy: The system achieved an average F1 score of 0.94 for patient matching across eight clinical domains, indicating a very high level of precision and recall.
  • Reliable Normalization: The agreement between the automated system and manual expert annotation was high, with a Cohen’s Kappa of 0.82.
  • Scale: The pipeline successfully normalized 367 unique eligibility attributes, demonstrating its ability to handle the "long tail" of complex medical requirements.

Limitations and Future Work

  • The Biomarker Challenge: The authors noted that EHRs often lack structured data for specific genetic biomarkers. Expanding how biomarker measurements are captured in real-world data is essential for the next generation of precision medicine trials.
  • Manual Curation: While the matching is automated, the initial normalization of some attributes still required manual effort. The authors suggest that future work could leverage Large Language Models (LLMs) or deep-learning NLP techniques (like CRF or Bi-LSTM) to automate the extraction of attributes from clinical notes.
  • Trial Scope: This study focused on only one "arm" of clinical trials; future iterations aim to include all trial arms to provide a more comprehensive matching service.

Summary updated

Abstract

Abstract The use of electronic health records (EHRs) holds the potential to enhance clinical trial activities. However, the identification of eligible patients within EHRs presents considerable challenges. We aimed to develop a pipeline for phenotyping eligibility criteria, enabling the identification of patients from EHRs with clinical characteristics that match those criteria. We utilized clinical trial eligibility criteria and patient EHRs from the Mount Sinai Database. The criteria and EHR data were normalized using national standard terminologies and in-house databases, facilitating computability and queryability. The pipeline employed rule-based pattern recognition and manual annotation. Our pipeline normalized 367 out of 640 unique eligibility criteria attributes, covering various medical conditions including non-small cell lung cancer, small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, Crohn’s disease, non-alcoholic steatohepatitis, and sickle cell anemia. 174 were encoded with standard terminologies and 193 were normalized using the in-house reference tables. The agreement between automated and manual normalization was high (Cohen’s Kappa = 0.82), and patient matching demonstrated a 0.94 F1 score. Our system has proven effective on EHRs from multiple institutions, showing broad applicability and promising improved clinical trial processes, leading to better patient selection, and enhanced clinical research outcomes.
BibTeX
@misc{lee2024establishing,
  author = {K. Lee and Yun Mai and Z. Liu and Kalpana Raja and Tomi Jun and Meng Ma and Tao Wang and Lei Ai and Ediz S. Calay and William Oh and Eric E. Schadt and Xingzhu Wang},
  title = {{Establishing the Automatic Identification of Clinical Trial Cohorts from Electronic Health Records by Matching Normalized Eligibility Criteria and Patient Clinical Characteristics}},
  year = {2024},
  doi = {10.1101/2024.02.28.24303396},
  howpublished = {medRxiv},
}
Figure