TL;DR
This paper introduces a novel framework leveraging Large Language Models (LLMs) to automatically detect Pulmonary Embolism as an Adverse Event (PEAE) from unstructured clinical notes in Electronic Medical Records (EMRs). The study demonstrates that LLMs can achieve high sensitivity and specificity, offering a promising avenue for improving patient safety monitoring and clinical research, though further refinement is needed to address false positives.
What Problem Does This Paper Address?
Identifying adverse events (AEs) in healthcare is critical for patient safety, quality improvement, and regulatory compliance. However, accurately detecting these events from Electronic Medical Records (EMRs) is a significant challenge. This is due to the low incidence of many AEs, the variability in how clinicians document them, and the complex, unstructured nature of clinical narratives. Pulmonary Embolism as an Adverse Event (PEAE) is particularly difficult to identify because it requires distinguishing hospital-acquired cases from community-acquired ones, establishing causality with medical procedures, and analyzing temporal relationships within patient histories. Traditional methods, relying on structured data like ICD codes or rigid keyword searches, often miss nuances and struggle with the complexity of clinical documentation.
Methods
The researchers developed a three-module LLM-based framework for PEAE detection.
- Evidence Extraction Module: This module tackles the challenge of processing lengthy clinical narratives by first "chunking" the text into smaller segments (paragraphs, sentences). It then employs two strategies to select relevant chunks for LLM analysis:
- Keyword-based filtering: Uses predefined PE-related keywords to identify relevant text.
- Semantic similarity-based filtering: Leverages embedding models to find text semantically similar to known PEAE descriptions, aiming to capture information even without exact keyword matches. This involved training an embedding model on a custom triplet dataset.
- Discharge Information Extraction Module: This module focuses on extracting key information from discharge summaries, which contain a comprehensive overview of a patient's hospital stay. It specifically targets six sections: reason for hospitalization, significant findings, procedures/treatments, medical history, discharge condition, and patient/family instructions.
- PEAE Detection Module: This module integrates information from the previous two modules to make a final determination. It constructs structured prompts for LLMs, feeding them the extracted evidence and discharge information. The LLMs are then tasked with a binary classification ("yes" or "no") followed by a justification.
The study evaluated four open-source LLMs (Llama3, Mistral-7B, Gemma, and Phi-3) using various configurations, including the inclusion or exclusion of discharge summaries and the use of keyword-based versus semantic similarity-based retrieval.
Datasets
The study utilized a retrospective chart review cohort of adult patients (aged 18-100) admitted to tertiary-care hospitals in Calgary, Alberta, Canada, between 2017 and 2022. A total of 10,066 patient records were reviewed. The chart review was conducted by six registered nurses following a predefined protocol, identifying 40 cases of PEAE, resulting in a prevalence of 0.4%. No new benchmark datasets were introduced; the study used existing EMR data.
Key Insights and Ideas
- LLMs show strong potential for AE detection: The study demonstrates that LLMs can effectively identify PEAE with high sensitivity and specificity, outperforming traditional rule-based methods in some aspects.
- Keyword-based filtering is surprisingly effective: Despite the advancements in semantic understanding, a straightforward keyword-based chunking approach, when combined with domain knowledge, proved more effective in selecting relevant text for PEAE detection than semantic similarity-based methods.
- Discharge summaries are crucial: Including information from discharge summaries significantly improved the performance metrics of the LLM framework, highlighting their value in providing comprehensive clinical context.
- Low prevalence is a major hurdle: The low incidence of PEAE (0.4%) in the dataset led to a relatively high number of false positives, resulting in a lower Positive Predictive Value (PPV). This means LLMs are good at ruling out non-PEAE cases but cannot be solely relied upon for definitive PEAE identification.
- Yearly trends are more robust: For population-level surveillance, the LLM framework showed strong correlations with actual PEAE trends at a yearly granularity, but performance became more variable at finer granularities (quarterly, monthly) due to data sparsity.
Key Results
- All four LLMs demonstrated high sensitivity (87.5-100%) and specificity (94.9-98.9%).
- Gemma, using keyword-based retrieval with discharge summaries, achieved the highest F1-score of 28.11%, with 87.5% sensitivity and 98.4% specificity.
- Keyword-based filtering reduced the median chunks per patient from 789 to 310, while semantic filtering further reduced this to 9 chunks.
- Including discharge summaries generally improved F1 scores across models.
- For population-level surveillance, all models showed strong correlation with actual PEAE trends at yearly granularity (r=0.92-0.99), with Llama3 achieving the highest correlation (0.988).
- The ICD-10-based method had a higher PPV (37.5%) but much lower sensitivity (14.52%) compared to the LLMs.
Limitations and Future Work
The authors acknowledge several limitations:
- False Positives: The most significant issue was the high rate of false positives, stemming from challenges in contextual misinterpretation, terminology confusion, and limitations in compressing complex medical information.
- Temporal Confusion: LLMs sometimes struggled to differentiate historical medical events from current conditions.
- Diagnostic Uncertainty: Suspected conditions were sometimes misclassified as confirmed diagnoses.
- Implicit Cases: The framework relies on explicitly documented information, potentially missing cases where PEAE is implied but not directly stated.
- Preprocessing Limitations: Technical limitations in chunking algorithms can lead to semantically inconsistent segments.
- Discharge Summary Extraction: The module occasionally overemphasized primary diagnoses, potentially omitting crucial secondary information.
Future work should focus on:
- Improving LLMs' contextual understanding and medical terminology interpretation.
- Exploring advanced prompting techniques (e.g., chain-of-thought, few-shot prompting) to enhance precision.
- Developing more robust methods for handling data imbalance and low-incidence events.
- Further refining preprocessing and information extraction modules.
- Expanding the framework to detect other adverse events beyond PEAE.
Summary updated