Combining Clinician Expertise with Prompt Engineering enhances Small Language Models Reliability for Cancer Entity Recognition in Electronic Health Records

Federica Corso, V. Peppoloni, Laura Mazzeo, G. Leone, Luana Passos, V. Miskovic, Justin Armanini, A. Ferrarin, Isabella C. Wiest, F M Wolf, Giulia Montelatici, Rebecca Romanò, Paolo Ambrosini, Tommaso Capoccia, Stefano Natangelo, Simone Rota, Paola Andena, Massimo Ponti, Alessandra Russo, G Stasi, L. Provenzano, A. Spagnoletti, Marco Meazza Prina, Chiara Cavalli, Claudia Giani, Roberta Serino, Michele Borracino, C. Bonalume, Rosa Maria Di Mauro, C. Agosta, Andra Diana Dumitrascu, Giorgia Di Liberti, Giulia Corrao, Teresa Beninato, Monica Ganzinelli, Mario Occhipinti, Marta Brambilla, Claudia Proto, Jakob Nikolas Kather, Alessandra Pedrocchi, Filippo de Braud, Giuseppe Lo Russo, Paolo Baili, Arsela PrelajmedRxiv 2025 · DOI

TL;DR

This paper explores how Small Language Models (SLMs) can reliably extract crucial cancer information from unstructured Electronic Health Records (EHRs). By combining clinician expertise with clever prompt engineering, they demonstrate that general-purpose SLMs like LLaMA 3.1 8B can outperform specialized medical models, especially when using few-shot prompting and native language inputs, paving the way for more efficient and private healthcare data analysis.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of real-world data, vital for understanding complex diseases like cancer. However, this data is often locked away in unstructured text, making it difficult and time-consuming to extract meaningful information. The variability in clinical language, semantic complexity, and privacy concerns create significant hurdles. While Large Language Models (LLMs) show promise, deploying them for sensitive medical data raises privacy issues, and their effectiveness in extracting specific clinical entities from diverse EHRs, especially in languages other than English, needs thorough investigation. This paper tackles the challenge of reliably extracting key cancer-related information from Italian EHRs using privacy-preserving SLMs, exploring how different prompting strategies and clinician expertise impact performance.

Methods

The researchers evaluated four locally deployable SLMs: LLaMA 3.1 8B, Mistral 7B (general-purpose), MedLLaMA 2 7B, and BioMistral 7B (biomedical-specialized). They tested three prompting strategies: zero-shot (no examples), few-shot (a few generic examples), and annotated few-shot (examples from their specific dataset). The study also compared performance using both English and Italian prompts. A key aspect was involving clinicians with varying levels of experience (experienced oncologists, residents, and medical students) to assess how their expertise influences prompt design and the consistency of extracted information. The extraction process was guided by a JSON schema to ensure structured output.

Datasets

The study utilized a real-world dataset from the APOLLO 11 trial, comprising 897 non-small cell lung cancer (NSCLC) patients. This dataset was divided into a development set (IO cohort, n=711) and a validation set (TKI cohort, n=186). The ground truth for evaluation was derived from the REDCap electronic Case Report Form for APOLLO11, with manual revision by oncologists and data scientists.

Key Insights and Ideas

  • Generalists Trump Specialists (Sometimes): Surprisingly, general-purpose models like LLaMA 3.1 8B often outperformed specialized biomedical models, particularly for simpler, binary extraction tasks. This suggests that broad reasoning capabilities can be highly effective even in specialized domains.
  • Prompting is Paramount: The way you ask matters. Few-shot prompting, especially with annotated examples from the target dataset, significantly boosted accuracy and reduced "hallucinations" (incorrectly generated information).
  • Language Matters: Using prompts in the native language (Italian, in this case) led to better extraction performance for several complex or implicitly expressed clinical variables, highlighting the importance of linguistic alignment.
  • Expertise Bridges the Gap: Clinical expertise plays a crucial role. While experienced clinicians showed more consistent annotations, few-shot prompting seemed to help less experienced users (students) improve their consistency, suggesting a scaffolding effect.
  • TNM Staging Remains a Hurdle: Extracting TNM staging information proved challenging for SLMs, likely due to its often implicit and descriptive nature in EHRs, requiring complex interpretation beyond simple keyword matching.

Key Results

  • LLaMA 3.1 8B achieved the highest average accuracy (0.81) and F1-score (0.62) across all models and prompting strategies, significantly outperforming Mistral 7B, BioMistral 7B, and MedLLaMA 2 7B.
  • Few-shot prompting with annotated data consistently improved performance across most features and models.
  • Italian prompts generally led to better extraction for features like smoking history and bone metastases compared to English prompts.
  • Clinical expertise significantly impacted annotation agreement, with experienced clinicians showing higher consistency. Few-shot prompting improved agreement among students.
  • Binary features were extracted more reliably than multiclass variables.

Limitations and Future Work

The authors acknowledge that extracting TNM staging remains a significant challenge, suggesting that further refinement through fine-tuning or Retrieval Augmented Generation (RAG) might be necessary. They also note that their study on clinical expertise involved a small cohort of oncologists, and future work should involve a larger and more diverse group of healthcare professionals. The study also plans to extend the set of extracted clinical features to include medications, timelines, side effects, and comorbidities. The findings highlight the need for hybrid systems that combine SLMs with expert input and for aligning clinical documentation practices with SLM capabilities.

Summary updated

Abstract

Real-world data (RWD), largely stored in unstructured electronic health records (EHRs), are critical for understanding complex diseases like cancer. However, extracting structured information from these narratives is challenging due to linguistic variability, semantic complexity, and privacy concerns. This study evaluates the performance of four locally deployable and small language models (SLMs), LLaMA, Mistral, BioMistral, and MedLLaMA, for information extraction (IE) from Italian EHRs within the APOLLO 11 trial on non-small cell lung cancer (NSCLC). We examined three prompting strategies (zero-shot, few-shot, and annotated few-shot) across English and Italian, involving clinicians with varying expertise to assess prompt design's impact on accuracy. Results show that general-purpose models (e.g., LLaMA 3.1 8B) outperform biomedical models in most tasks, particularly in extracting binary features. Multiclass variables such as TNM staging, PD-L1, and ECOG were more difficult due to implicit language and lack of standardization. Few-shot prompting and native-language inputs significantly improved performance and reduced hallucinations. Clinical expertise enhanced consistency in annotation, particularly among students using annotated examples. The study confirms that privacy-preserving SLMs can be deployed locally for efficient and secure cancer data extraction. Findings highlight the need for hybrid systems combining SLMs with expert input and underline the importance of aligning clinical documentation practices with SLM capabilities. This is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows.
BibTeX
@misc{corso2025combining,
  author = {Federica Corso and V. Peppoloni and Laura Mazzeo and G. Leone and Luana Passos and V. Miskovic and Justin Armanini and A. Ferrarin and Isabella C. Wiest and F M Wolf and Giulia Montelatici and Rebecca Romanò and Paolo Ambrosini and Tommaso Capoccia and Stefano Natangelo and Simone Rota and Paola Andena and Massimo Ponti and Alessandra Russo and G Stasi and L. Provenzano and A. Spagnoletti and Marco Meazza Prina and Chiara Cavalli and Claudia Giani and Roberta Serino and Michele Borracino and C. Bonalume and Rosa Maria Di Mauro and C. Agosta and Andra Diana Dumitrascu and Giorgia Di Liberti and Giulia Corrao and Teresa Beninato and Monica Ganzinelli and Mario Occhipinti and Marta Brambilla and Claudia Proto and Jakob Nikolas Kather and Alessandra Pedrocchi and Filippo de Braud and Giuseppe Lo Russo and Paolo Baili and Arsela Prelaj},
  title = {{Combining Clinician Expertise with Prompt Engineering enhances Small Language Models Reliability for Cancer Entity Recognition in Electronic Health Records}},
  year = {2025},
  doi = {10.1101/2025.10.16.25337917},
  howpublished = {medRxiv},
}
Figure