TL;DR
This paper explores how Small Language Models (SLMs) can reliably extract crucial cancer information from unstructured Electronic Health Records (EHRs). By combining clinician expertise with clever prompt engineering, they demonstrate that general-purpose SLMs like LLaMA 3.1 8B can outperform specialized medical models, especially when using few-shot prompting and native language inputs, paving the way for more efficient and private healthcare data analysis.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) are a treasure trove of real-world data, vital for understanding complex diseases like cancer. However, this data is often locked away in unstructured text, making it difficult and time-consuming to extract meaningful information. The variability in clinical language, semantic complexity, and privacy concerns create significant hurdles. While Large Language Models (LLMs) show promise, deploying them for sensitive medical data raises privacy issues, and their effectiveness in extracting specific clinical entities from diverse EHRs, especially in languages other than English, needs thorough investigation. This paper tackles the challenge of reliably extracting key cancer-related information from Italian EHRs using privacy-preserving SLMs, exploring how different prompting strategies and clinician expertise impact performance.
Methods
The researchers evaluated four locally deployable SLMs: LLaMA 3.1 8B, Mistral 7B (general-purpose), MedLLaMA 2 7B, and BioMistral 7B (biomedical-specialized). They tested three prompting strategies: zero-shot (no examples), few-shot (a few generic examples), and annotated few-shot (examples from their specific dataset). The study also compared performance using both English and Italian prompts. A key aspect was involving clinicians with varying levels of experience (experienced oncologists, residents, and medical students) to assess how their expertise influences prompt design and the consistency of extracted information. The extraction process was guided by a JSON schema to ensure structured output.
Datasets
The study utilized a real-world dataset from the APOLLO 11 trial, comprising 897 non-small cell lung cancer (NSCLC) patients. This dataset was divided into a development set (IO cohort, n=711) and a validation set (TKI cohort, n=186). The ground truth for evaluation was derived from the REDCap electronic Case Report Form for APOLLO11, with manual revision by oncologists and data scientists.
Key Insights and Ideas
- Generalists Trump Specialists (Sometimes): Surprisingly, general-purpose models like LLaMA 3.1 8B often outperformed specialized biomedical models, particularly for simpler, binary extraction tasks. This suggests that broad reasoning capabilities can be highly effective even in specialized domains.
- Prompting is Paramount: The way you ask matters. Few-shot prompting, especially with annotated examples from the target dataset, significantly boosted accuracy and reduced "hallucinations" (incorrectly generated information).
- Language Matters: Using prompts in the native language (Italian, in this case) led to better extraction performance for several complex or implicitly expressed clinical variables, highlighting the importance of linguistic alignment.
- Expertise Bridges the Gap: Clinical expertise plays a crucial role. While experienced clinicians showed more consistent annotations, few-shot prompting seemed to help less experienced users (students) improve their consistency, suggesting a scaffolding effect.
- TNM Staging Remains a Hurdle: Extracting TNM staging information proved challenging for SLMs, likely due to its often implicit and descriptive nature in EHRs, requiring complex interpretation beyond simple keyword matching.
Key Results
- LLaMA 3.1 8B achieved the highest average accuracy (0.81) and F1-score (0.62) across all models and prompting strategies, significantly outperforming Mistral 7B, BioMistral 7B, and MedLLaMA 2 7B.
- Few-shot prompting with annotated data consistently improved performance across most features and models.
- Italian prompts generally led to better extraction for features like smoking history and bone metastases compared to English prompts.
- Clinical expertise significantly impacted annotation agreement, with experienced clinicians showing higher consistency. Few-shot prompting improved agreement among students.
- Binary features were extracted more reliably than multiclass variables.
Limitations and Future Work
The authors acknowledge that extracting TNM staging remains a significant challenge, suggesting that further refinement through fine-tuning or Retrieval Augmented Generation (RAG) might be necessary. They also note that their study on clinical expertise involved a small cohort of oncologists, and future work should involve a larger and more diverse group of healthcare professionals. The study also plans to extend the set of extracted clinical features to include medications, timelines, side effects, and comorbidities. The findings highlight the need for hybrid systems that combine SLMs with expert input and for aligning clinical documentation practices with SLM capabilities.
Summary updated