Abstract
LLMs are effective at extracting functional biomarkers from EHR data and broadly generalizable across variable note-taking practices and institutions. Individual verbal and ambulatory ability were accurately extracted, supporting the method's ability to streamline workflows by offering automated, efficient data extraction for patient care and research. Future studies are needed to extend this methodology to additional populations and to demonstrate more granular functional data classification.
Introduction
In the management of neurodevelopmental disorders (IDDs) and cerebral palsy (CP), tracking functional status—specifically a patient’s ability to walk independently or communicate verbally—is essential for clinical decision-making and longitudinal research. Traditionally, this data is captured via standardized assessments such as surveys and clinical tests. While valuable, these assessments represent a "double-edged sword": they provide high-quality data but impose a significant administrative burden on providers and families. This often results in missing data or infrequent snapshots that fail to capture the nuances of patient progression.
Much of this functional information is already present within the "free-text" clinical notes of Electronic Health Records (EHRs). However, because this data is unstructured and varies significantly across institutions and providers, it has historically been difficult to aggregate. This study investigates the utility of modern Large Language Models (LLMs) to bridge this gap by accurately and cost-effectively extracting functional biomarkers from real-world clinical documentation.
Methodology
The researchers developed an automated pipeline utilizing three OpenAI models: GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4 Omni. The workflow was designed to process raw EHR clinical notes and generate both note-level and individual-level predictions regarding ambulatory and verbal abilities.
Iterative Prompt Engineering
A core component of the methodology was the development of a sophisticated prompting strategy. The researchers assigned the LLM a specific persona—a detail-oriented medical physician—to set the context for the task. They provided strict, clinical definitions for "walking" and "using words" to minimize model ambiguity.
The team tested two primary prompting styles:
- Multi-Class Prompting (MCP): Allowed the model to respond with "yes," "no," or "unknown."
- Binary Class Prompting (BCP): Forced a "yes" or "no" determination.
Aggregation and Cost Analysis
To move from individual notes to a patient-level status, the team implemented an aggregation logic. If the majority of a patient's notes indicated a specific ability, that became the final label. Additionally, a formal cost analysis was conducted to evaluate the economic feasibility of deploying such a system at scale.
Datasets and Ground Truth
The study evaluated the pipeline across two distinct cohorts to ensure generalizability:
- National Brain Gene Registry (BGR): A multi-institutional dataset from 12 academic centers focused on rare neurogenetic disorders, containing 3,245 notes from 125 individuals.
- Cerebral Palsy (CP) Clinic Cohort: A single-institution dataset from St. Louis Children’s Hospital, comprising 5,462 notes from 260 individuals.
LLM predictions were validated against "gold-standard" clinical scales, including the Gross Motor Function Classification System (GMFCS), the Viking Speech Scale (VSS), and the Communication Function Classification System (CFCS).
Results
Performance Metrics
The models demonstrated high accuracy in identifying ambulatory status, achieving weighted-F1 scores greater than 0.90 across both cohorts. Performance for verbal ability was excellent in the BGR cohort (F1 ~0.92 with GPT-4o). However, performance was lower in the CP cohort (F1 ~0.68). This discrepancy was attributed to the complexity of the CP ground-truth scales (VSS/CFCS), which measure speech effectiveness and quality rather than the simple presence of words.
Model Comparison and Cost
GPT-4 Turbo and GPT-4 Omni consistently outperformed GPT-3.5 Turbo, particularly when processing longer notes and following complex instructions. The total cost to analyze nearly 9,000 notes across all three models was $460.73. For a single study using a modern model like GPT-4o, the cost per patient is negligible compared to the manual labor required for clinical abstraction.
Discussion and Future Directions
The study highlighted that certain note types are more informative than others; Physical Therapy (PT) notes were primary for ambulatory data, while Psychology and Speech Therapy notes were best for verbal status. Notably, the pipeline showed strong zero-shot generalizability, performing well across 12 different institutions without requiring fine-tuning on specific hospital templates.
Limitations
The BGR cohort lacked demographic diversity, which may limit the findings' applicability to populations with different dialects or note-taking styles. Furthermore, the current study focused on binary classifications. Future research will aim to extract more granular data, such as specific GMFCS levels (1–5), and explore open-source models (e.g., Llama or Mixtral) to enhance data privacy and further reduce costs.
Summary updated