TL;DR
This paper introduces two novel datasets, SYNUR and SIMORD, for structured extraction of clinical information from nurse dictations and doctor-patient conversations, respectively. By evaluating various large language models (LLMs) on these datasets, the authors demonstrate the potential of LLMs to significantly reduce the documentation burden on healthcare providers, paving the way for more efficient clinical workflows.
What Problem Does This Paper Address?
Healthcare professionals spend a significant amount of time on documentation, which detracts from direct patient care. While Large Language Models (LLMs) have shown promise in various Natural Language Processing (NLP) tasks, two critical clinical applications remain underexplored due to data scarcity and sensitivity: structuring nurse dictations into organized observations and extracting medical orders from lengthy doctor-patient consultations. Existing benchmarks often fail to capture the complexities of real-world clinical data, such as long, disfluent speech and the need to integrate information into existing electronic health record (EHR) structures. This paper aims to bridge this gap by developing new datasets and evaluating LLM performance on these challenging tasks.
Methods
The paper tackles two main tasks:
- Nursing Observation Extraction: The authors propose a three-step pipeline for this task:
- Segmentation: The LLM breaks down long nurse dictations into medically coherent segments.
- RAG Filtering: A Retrieval-Augmented Generation (RAG) approach filters potential rows from a flowsheet schema based on cosine similarity with the current segment, addressing the context window limitations of LLMs.
-
Extraction: The LLM extracts relevant clinical observations from the filtered segments and canonicalizes them according to the reduced schema. They also developed an agentic data generation pipeline (SYNUR) to create synthetic nurse dictations.
-
Medical Order Extraction: This task is framed as a reading comprehension problem for LLMs. The authors provide a detailed prompt that guides LLMs to extract four key attributes for each order: description, type (medication, lab, follow-up, imaging), reason, and provenance (conversation line numbers).
Datasets
The paper introduces two new datasets:
- SYNUR (Synthetic NURsing dataset): This is the first open-source synthetic dataset for extracting structured observations from spoken nurse dictations. It was created using an LLM-driven pipeline that involved expert review and annotation, resulting in 223 synthetic dictations with 3000 observations.
- SIMORD (SIMulated ORDer dataset): This is the first open-source dataset for extracting medical orders from doctor-patient simulated transcripts. It was created by medically trained annotators who produced gold-standard orders for conversations from existing datasets (ACI-Bench and PriMock57), resulting in 64 training samples and a test set.
Key Insights and Ideas
- LLMs can handle real-world clinical complexity: Despite challenges like long contexts and disfluent speech, LLMs show promising performance on these complex, unstructured clinical data tasks.
- Synthetic data generation is viable for clinical NLP: The SYNUR dataset demonstrates that LLMs can be leveraged to create realistic, privacy-preserving synthetic clinical data, mitigating the scarcity and sensitivity issues of real-world data.
- Contextualization is key for nursing observations: The RAG filtering step in the nursing observation extraction pipeline is crucial for handling the overflow of information in typical EHR flowsheets.
- Prompt engineering matters: The performance of LLMs is significantly influenced by the quality and detail of the prompts, especially when dealing with nuanced extraction tasks.
- Open-weight models show potential but lag behind closed-weight models: While open-weight models are improving, closed-weight models, particularly GPT-4.1, generally achieve better performance on these clinical tasks, though some open-weight models show competitive results with fine-tuning or few-shot learning.
Key Results
- Nursing Observation Extraction: GPT-4.1 consistently outperformed other models across proprietary datasets, while GPT-40 performed best on the synthetic SYNUR dataset. Including examples in the prompt led to higher F1 scores for all models.
- Medical Order Extraction: No single closed-weight model consistently outperformed others across all metrics. GPT-40 achieved high scores in Description and Type in zero-shot settings, while 01-mini excelled in Provenance. Introducing examples improved performance, especially for free-text fields like Description and Reason. Open-weight models showed varied performance, with some achieving competitive results on specific metrics with few-shot learning.
Limitations and Future Work
- Synthetic data limitations: SYNUR, while valuable, cannot fully capture the richness and variability of real clinical language.
- Dataset size: The SIMORD dataset is relatively small, which may constrain its applicability for fine-tuning.
- Annotation variability: Despite high inter-annotator agreement, some variability exists in annotations due to informal phrasing and span boundary ambiguities.
- Future work: The authors suggest expanding the SIMORD dataset, integrating more sophisticated RAG approaches, and leveraging real nurse dictations to build more comprehensive nursing ontologies. They also highlight the need for robust parsing mechanisms for LLM outputs, particularly for open-weight models.
Summary updated