TL;DR
This paper introduces a novel framework that allows smaller, locally run AI models to leverage the advanced reasoning capabilities of powerful, proprietary large language models (LLMs) for generating medical progress notes. By intelligently distilling relevant clinical information without exposing sensitive patient data, their approach significantly improves the performance of local models, addressing a critical bottleneck in medical AI.
What Problem Does This Paper Address?
The generation of medical progress notes is a time-consuming and burdensome task for physicians, contributing to burnout. While large language models (LLMs) show great promise in automating this, powerful proprietary LLMs are typically accessed via web APIs, raising privacy concerns for sensitive electronic health records (EHRs). Conversely, locally run models, while more privacy-friendly, are often smaller and lack the sophisticated reasoning and contextual understanding of their API-based counterparts. This creates a dilemma: how to harness the power of cutting-edge LLMs for medical tasks without compromising patient privacy or relying on resource-intensive local deployments. This paper tackles the specific challenge of generating progress notes by bridging this gap, enabling local models to benefit from external LLM intelligence in a privacy-preserving manner.
Methods
The core of the proposed framework involves a two-stage process. First, a proprietary web-based LLM (specifically, Anthropic's Claude 3 Opus) is used to distill relevant information. This stage focuses on identifying key clinical concepts within the "Prior" progress note and then querying the LLM with these concepts along with a list of available structured chart data labels (but not the actual patient data values). The LLM's task is to select which of these data labels are most relevant for assessing the patient's progress for a given problem.
The second stage involves a locally run model. The selected, distilled data labels are then processed to summarize the interim structured chart data. This summarized data, along with the "Prior" progress note and the identified problem, is fed into a locally run LLM (such as Mixtral 8x7B). This local model then generates the "Next" progress note. This approach ensures that sensitive patient data never leaves the local environment, while still benefiting from the external LLM's ability to identify crucial clinical indicators.
Datasets
The study utilizes the MIMIC-III database, a publicly accessible dataset collected from an intensive care unit (ICU) setting. For their evaluation, they sampled pairs of subsequent progress notes from ICU admissions, ensuring no other notes were documented in between and that non-empty structured chart data was available. This resulted in 7089 annotation instances, from which 100 instances were randomly sampled for quantitative evaluation.
Key Insights and Ideas
- Privacy-Preserving Knowledge Distillation: The paper demonstrates a practical method for extracting valuable clinical insights from powerful proprietary LLMs without compromising patient data privacy. This is achieved by carefully designing prompts that focus on data labels rather than raw patient values.
- Addressing the Bottleneck: The framework directly tackles the challenge of processing the vast amount of interim structured chart data that often overwhelms local models. By using an external LLM to filter and summarize this data, it creates a manageable input for the local model.
- Problem-Centric Approach: Segmenting the progress note generation task by individual clinical problems (rather than processing the entire note at once) allows for more focused and effective knowledge extraction and generation.
- Synergy of Local and Cloud Models: The research highlights the potential for a hybrid approach where the strengths of both locally run models (privacy, control) and cloud-based LLMs (reasoning, knowledge) can be combined to achieve superior results.
Key Results
The proposed framework significantly improved the performance of the Mixtral 8x7B model on progress note generation. Specifically, it achieved a 4.6-point increase on the ROUGE-Lsum metric and a 7.56-point increase on the MEDCON F1 score. Ablation studies showed that using problem segments and distilled structured chart data labels, as opposed to the entire note and all available data, led to improvements across all models evaluated. The best-performing model, Mixtral, demonstrated strong capabilities in capturing overall status changes and did not exhibit hallucinations in manual evaluations.
Limitations and Future Work
The authors acknowledge that their framework is currently unable to capture new problems that may have emerged between progress notes. They also note the challenge of incorporating information beyond structured chart data, such as radiology reports, due to the limited context window of existing on-premise models. Future work could explore methods to address these limitations, potentially by developing more sophisticated context management techniques or integrating multimodal data sources.
Summary updated